{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Natural Language Processing with Deep Learning Techniques\n",
    "\n",
    "This demo will walk through two networks - a bi-directional LSTM and a temporal CNN. The bi-directional LSTM is trained with a pre-trained embedding that it then improves upon. For simplicity sake, both networks are shown using the Keras framework.\n",
    "\n",
    "The dataset used for the demonstration is the Ditigal Music reviews from: http://jmcauley.ucsd.edu/data/amazon/ and the pre-trained word embedding is Google News' 300d vector which can be downloaded from https://code.google.com/archive/p/word2vec/. Please download the dataset and model from those locations. The terms and conditions of the data set license apply. Intel does not grant any rights to the data files. The Amazon Review Dataset was published in the following papers:\n",
    "\n",
    "Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering\n",
    "R. He, J. McAuley\n",
    "WWW, 2016\n",
    "http://cseweb.ucsd.edu/~jmcauley/pdfs/www16a.pdf\n",
    "\n",
    "Image-based recommendations on styles and substitutes\n",
    "J. McAuley, C. Targett, J. Shi, A. van den Hengel\n",
    "SIGIR, 2015\n",
    "http://cseweb.ucsd.edu/~jmcauley/pdfs/sigir15.pdf\n",
    "\n",
    "This demonstration uses the Matplotlib library \"Copyright (c) 2012- Matplotlib Development Team; All Rights Reserved\"\n",
    "\n",
    "In this example we are taking the most positive and most negative reviews from that dataset, and using simple normalization techniques. More complex techniques and larger datasets can be applied to get better results as currently the networks are overfitting in the examples. We will discuss some strategies to mitigate overfitting.\n",
    "\n",
    "Everything can be run on a laptop, Python 3.6 is used, though it should all work in Python 2.7 as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd \n",
    "import json\n",
    "import gensim\n",
    "import re\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.optimizers import SGD\n",
    "from keras.models import Sequential, Model\n",
    "from keras.layers.core import Dense, Dropout, Activation, Flatten\n",
    "from keras.layers import Input, Bidirectional, LSTM\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers.convolutional import Conv1D, MaxPooling1D\n",
    "\n",
    "\n",
    "!pip install matplotlib\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Change these to match your file paths :)\n",
    "filename = '/data_path/reviews_Digital_Music_5.json' #64,706 reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_path = \"model_path/GoogleNews-vectors-negative300.bin\"\n",
    "word_vector_model = gensim.models.KeyedVectors.load_word2vec_format(model_path, binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def normalize(txt, vocab=None, replace_char=' ',\n",
    "                max_length=300, pad_out=False,\n",
    "                to_lower=True, reverse = False,\n",
    "                truncate_left=False, encoding=None,\n",
    "                letters_only=False):\n",
    "  \n",
    "    txt = txt.split()\n",
    "    # Remove HTML\n",
    "    # This will keep characters and other symbols\n",
    "    txt = [re.sub(r'http:.*', '', r) for r in txt]\n",
    "    txt = [re.sub(r'https:.*', '', r) for r in txt]\n",
    "    \n",
    "    txt = ( \" \".join(txt))\n",
    "    # Remove non-emoticon punctuation and numbers\n",
    "    txt = re.sub(\"[.,!0-9]\", \" \", txt)\n",
    "    if letters_only: \n",
    "        txt = re.sub(\"[^a-zA-Z]\", \" \", txt)\n",
    "    txt = \" \".join(txt.split())\n",
    "    # store length for multiple comparisons\n",
    "    txt_len = len(txt)\n",
    "\n",
    "    if truncate_left:\n",
    "        txt = txt[-max_length:]\n",
    "    else:\n",
    "        txt = txt[:max_length]\n",
    "    # change case\n",
    "    if to_lower:\n",
    "        txt = txt.lower()\n",
    "    # Reverse order\n",
    "    if reverse:\n",
    "        txt = txt[::-1]\n",
    "    # replace chars\n",
    "    if vocab is not None:\n",
    "        txt = ''.join([c if c in vocab else replace_char for c in txt])\n",
    "    # re-encode text\n",
    "    if encoding is not None:\n",
    "        txt = txt.encode(encoding, errors=\"ignore\")\n",
    "    # pad out if needed\n",
    "    if pad_out and max_length>txt_len:\n",
    "        txt = txt + replace_char * (max_length - txt_len)\n",
    "    return txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this is a sentence with things :) and a link\n"
     ]
    }
   ],
   "source": [
    "# What does this normalization function look like?\n",
    "clean_text = normalize(\"This is A sentence. with things! 123 :) and a link https://gitub.com\")\n",
    "print(clean_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def balance(df):\n",
    "    print(\"Balancing the classes\")\n",
    "    type_counts = df['Sentiment'].value_counts()\n",
    "    min_count = min(type_counts.values)\n",
    "\n",
    "    balanced_df = None\n",
    "    for key in type_counts.keys():\n",
    "\n",
    "        df_sub = df[df['Sentiment']==key].sample(n=min_count, replace=False)\n",
    "        if balanced_df is not None:\n",
    "            balanced_df = balanced_df.append(df_sub)\n",
    "        else:\n",
    "            balanced_df = df_sub\n",
    "    return balanced_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of the larger Amazon datasts have to be loaded line by line as they are too large to be directly read into a Pandas DataFrame, but the smaller files may be read directly. I'm also changing the reviews into a sentiment, mapping the 1 stars as negative and 5 stars as positive. This function can be modified as desired, or you can keep the stars and use that as your prediction mechanism."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "good_columns = [\n",
    "    \"overall\", \n",
    "    \"reviewText\", \n",
    "    \"summary\"\n",
    "]\n",
    "\n",
    "def review_to_sentiment(review):\n",
    "    # Review is coming in as overall (the rating, reviewText, and summary)\n",
    "    # this then cleans the summary and review and gives it a positive or negative value\n",
    "    norm_text = normalize(review[2] + \" \" + review[1])\n",
    "    if review[0] ==5:\n",
    "        return ['positive', norm_text]\n",
    "    elif review[0] ==1:\n",
    "        return ['negative', norm_text]\n",
    "    else:\n",
    "        return ['other', norm_text]\n",
    "    \n",
    "\n",
    "\n",
    "data = []\n",
    "with open(filename, 'r') as f:\n",
    "    for line in f:\n",
    "        data_line = json.loads(line)\n",
    "        selected_row = []\n",
    "        for item in good_columns:\n",
    "            selected_row.append(data_line[item])\n",
    "        # as we read in, clean\n",
    "        data.append(review_to_sentiment(selected_row))\n",
    "        \n",
    "amazon = pd.DataFrame(data, columns=['Sentiment', 'clean_text'], dtype=str)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sentiment</th>\n",
       "      <th>clean_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>positive</td>\n",
       "      <td>enya's last great album it's hard to believe \"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>positive</td>\n",
       "      <td>enya at her most elegant a clasically-styled a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>positive</td>\n",
       "      <td>the best so far i never thought enya would rea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>positive</td>\n",
       "      <td>ireland produces good music this is the third ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>positive</td>\n",
       "      <td>dreamy unearthly magic continues with eire's o...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Sentiment                                         clean_text\n",
       "0  positive  enya's last great album it's hard to believe \"...\n",
       "1  positive  enya at her most elegant a clasically-styled a...\n",
       "2  positive  the best so far i never thought enya would rea...\n",
       "3  positive  ireland produces good music this is the third ...\n",
       "5  positive  dreamy unearthly magic continues with eire's o..."
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For this demo lets just keep one and five stars the others are marked 'other\n",
    "amazon = amazon[amazon['Sentiment'].isin(['positive', 'negative'])]\n",
    "amazon.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Balancing the classes\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "5582"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "balanced_amazon = balance(amazon)\n",
    "len(balanced_amazon)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Now go from the pandas into lists of text and labels\n",
    "\n",
    "text = balanced_amazon['clean_text'].values\n",
    "labels_0 = pd.get_dummies(balanced_amazon['Sentiment'])  # mapping of the labels with dummies (has headers)\n",
    "labels = labels_0.values # removes the headers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 1],\n",
       "       [0, 1],\n",
       "       [0, 1],\n",
       "       [0, 1],\n",
       "       [0, 1],\n",
       "       [0, 1],\n",
       "       [0, 1],\n",
       "       [0, 1],\n",
       "       [0, 1],\n",
       "       [0, 1]], dtype=uint8)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Perform the Train/test split\n",
    "X_train_, X_test_, Y_train_, Y_test_ = train_test_split(text,labels, test_size = 0.2, random_state = 42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0],\n",
       "       [0, 1],\n",
       "       [1, 0],\n",
       "       [0, 1],\n",
       "       [1, 0]], dtype=uint8)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# What does the data look like?\n",
    "# It is a one-hot encoding of the label, either positive or negative\n",
    "Y_train_[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'a true poet gil scott-heron is one of the greatest poets the world has ever produced his words still hit home very underrated when folks talk about lyrical genius but he has written songs that are timeless&amp;very much in your face&amp;mind the music is great as well solid production&amp;arrangemen'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_[42]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "### Now for a simple bidirectional LSTM algorithm we set our feature sizes and train a tokenizer\n",
    "# First we Tokenize and get the data into a form that the model can read - this is BoW\n",
    "# In this cell we are also going to define some of our hyperparameters\n",
    "\n",
    "max_fatures = 2000\n",
    "max_len=300\n",
    "batch_size = 32\n",
    "embed_dim = 300\n",
    "lstm_out = 140\n",
    "\n",
    "dense_out=len(labels[0]) #length of features\n",
    "tokenizer = Tokenizer(num_words=max_fatures, split=' ')\n",
    "tokenizer.fit_on_texts(X_train_)\n",
    "X_train = tokenizer.texts_to_sequences(X_train_)\n",
    "X_train = pad_sequences(X_train, maxlen=max_len, padding='post')\n",
    "\n",
    "X_test = tokenizer.texts_to_sequences(X_test_)\n",
    "X_test = pad_sequences(X_test, maxlen=max_len, padding='post')\n",
    "\n",
    "word_index = tokenizer.word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   3,  255, 1240,    5,   19,    6,    1,  193,    1,  223,   36,\n",
       "         69,  473,   26,  454,  102,  207,  614,   72,  869,   52,  790,\n",
       "        751,   48,  716,  528,   17,   28,   36,  626,   34,   12,   22,\n",
       "        791,  241,   72,   95,   11,   84,  479,  241,  408,    1,   29,\n",
       "          5,   42,   23,  114,  870,  235,  241,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0], dtype=int32)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now what does our data look like?\n",
    "# Tokenizer creates a BOW encoding, which is then going to be fed into our Embedding matrix\n",
    "# This will be used by the model to build up a word embedding\n",
    "X_train[42]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# What does a word vector look like?\n",
    "# Ahhhh, like a bunch of numbers\n",
    "word_vector_model.word_vec('hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prepare the embedding matrix\n"
     ]
    }
   ],
   "source": [
    "print('Prepare the embedding matrix')\n",
    "\n",
    "# prepare embedding matrix\n",
    "num_words = min(max_fatures, len(word_index))\n",
    "embedding_matrix = np.zeros((num_words, embed_dim))\n",
    "for word, i in word_index.items():\n",
    "    if i >= max_len:\n",
    "        continue\n",
    "    # words not found in embedding index will be all-zeros.\n",
    "    if word in word_vector_model.vocab:\n",
    "        embedding_matrix[i] = word_vector_model.word_vec(word)\n",
    "\n",
    "# load pre-trained word embeddings into an Embedding layer\n",
    "# note that we set trainable = True to fine tune the embeddings\n",
    "embedding_layer = Embedding(num_words,\n",
    "                            embed_dim,\n",
    "                            weights=[embedding_matrix],\n",
    "                            input_length=max_fatures,\n",
    "                            trainable=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.08007812,  0.10498047,  0.04980469,  0.0534668 , -0.06738281,\n",
       "       -0.12060547,  0.03515625, -0.11865234,  0.04394531,  0.03015137,\n",
       "       -0.05688477, -0.07617188,  0.01287842,  0.04980469, -0.08496094,\n",
       "       -0.06347656,  0.00628662, -0.04321289,  0.02026367,  0.01330566,\n",
       "       -0.01953125,  0.09277344, -0.171875  , -0.00131989,  0.06542969,\n",
       "        0.05834961, -0.08251953,  0.0859375 , -0.00318909,  0.05859375,\n",
       "       -0.03491211, -0.0123291 , -0.0480957 , -0.00302124,  0.05639648,\n",
       "        0.01495361, -0.07226562, -0.05224609,  0.09667969,  0.04296875,\n",
       "       -0.03540039, -0.07324219,  0.03271484, -0.06176758,  0.00787354,\n",
       "        0.0035553 , -0.00878906,  0.0390625 ,  0.03833008,  0.04443359,\n",
       "        0.06982422,  0.01263428, -0.00445557, -0.03320312, -0.04272461,\n",
       "        0.09765625, -0.02160645, -0.0378418 ,  0.01190186, -0.01391602,\n",
       "       -0.11328125,  0.09326172, -0.03930664, -0.11621094,  0.02331543,\n",
       "       -0.01599121,  0.02636719,  0.10742188, -0.00466919,  0.09619141,\n",
       "        0.0279541 , -0.05395508,  0.08544922, -0.03686523, -0.02026367,\n",
       "       -0.08544922,  0.125     ,  0.14453125,  0.0267334 ,  0.15039062,\n",
       "        0.05273438, -0.18652344,  0.08154297, -0.01062012, -0.03735352,\n",
       "       -0.07324219, -0.07519531,  0.03613281, -0.13183594,  0.00616455,\n",
       "        0.05078125,  0.04516602,  0.0100708 , -0.15039062, -0.06005859,\n",
       "        0.05761719, -0.00692749,  0.01586914, -0.0213623 ,  0.10351562,\n",
       "       -0.00029182, -0.046875  , -0.01635742, -0.07861328, -0.06933594,\n",
       "        0.01635742, -0.03149414, -0.01373291, -0.03662109, -0.08886719,\n",
       "       -0.0480957 , -0.01318359, -0.07177734,  0.00588989, -0.04614258,\n",
       "        0.03979492,  0.10058594, -0.04931641,  0.07568359,  0.03881836,\n",
       "       -0.16699219, -0.09619141, -0.10107422,  0.02905273, -0.05786133,\n",
       "       -0.01928711, -0.04296875, -0.08398438, -0.01989746,  0.05151367,\n",
       "        0.00848389, -0.03613281, -0.14941406, -0.01855469, -0.03637695,\n",
       "       -0.07666016, -0.03955078, -0.06152344, -0.02001953,  0.04150391,\n",
       "        0.03686523, -0.07226562,  0.00592041, -0.06298828,  0.00738525,\n",
       "       -0.01586914,  0.01611328, -0.01452637,  0.00772095,  0.10107422,\n",
       "       -0.00558472,  0.01428223, -0.07617188,  0.05639648, -0.01293945,\n",
       "        0.03063965, -0.02490234, -0.09863281,  0.0324707 , -0.02807617,\n",
       "       -0.08105469,  0.02062988,  0.01611328, -0.04199219, -0.03491211,\n",
       "       -0.03759766,  0.05493164,  0.01373291,  0.02685547, -0.05859375,\n",
       "       -0.07177734, -0.12011719, -0.02282715, -0.1640625 , -0.00361633,\n",
       "       -0.05981445,  0.07080078, -0.07714844,  0.05175781, -0.04296875,\n",
       "       -0.04833984,  0.0300293 , -0.06591797, -0.03173828, -0.04882812,\n",
       "       -0.03491211,  0.05883789, -0.01464844,  0.18066406,  0.05688477,\n",
       "        0.05249023,  0.05786133,  0.11669922,  0.05200195, -0.0534668 ,\n",
       "        0.01867676, -0.015625  ,  0.00576782, -0.07324219, -0.11621094,\n",
       "        0.04052734,  0.0625    , -0.04321289,  0.01055908,  0.02172852,\n",
       "        0.04248047,  0.03271484,  0.04418945,  0.05761719,  0.02612305,\n",
       "       -0.01831055, -0.02697754, -0.00674438,  0.00509644, -0.11621094,\n",
       "        0.00364685,  0.05761719, -0.05957031, -0.08837891,  0.0135498 ,\n",
       "        0.04541016, -0.04638672, -0.0177002 , -0.0625    ,  0.03442383,\n",
       "       -0.02416992,  0.03088379,  0.09570312,  0.07958984,  0.03930664,\n",
       "        0.0279541 , -0.0859375 ,  0.08105469,  0.06640625, -0.00041962,\n",
       "       -0.06933594,  0.03588867, -0.03417969,  0.04492188, -0.00772095,\n",
       "       -0.00741577, -0.04760742,  0.01397705, -0.09960938,  0.0246582 ,\n",
       "       -0.09960938,  0.11474609,  0.03173828,  0.02209473,  0.07226562,\n",
       "        0.03686523,  0.02563477,  0.01367188, -0.02734375,  0.00592041,\n",
       "       -0.06738281,  0.05053711, -0.02832031, -0.04516602, -0.01733398,\n",
       "        0.02111816,  0.03515625, -0.04296875,  0.06640625,  0.12207031,\n",
       "        0.12353516,  0.0039978 ,  0.04516602, -0.01855469,  0.04833984,\n",
       "        0.04516602,  0.08691406,  0.02941895,  0.03759766,  0.03442383,\n",
       "       -0.07373047, -0.0402832 , -0.14648438, -0.02441406, -0.01953125,\n",
       "        0.0065918 , -0.0018158 , -0.01092529,  0.09326172,  0.06542969,\n",
       "        0.01843262, -0.09326172, -0.01574707, -0.07128906, -0.08935547,\n",
       "       -0.07128906, -0.03015137, -0.01300049,  0.01635742, -0.01831055,\n",
       "        0.01483154,  0.00500488,  0.00366211,  0.04760742, -0.06884766])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding_matrix[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /Users/annabeth/anaconda/envs/py36/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1344: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n"
     ]
    }
   ],
   "source": [
    "# Define the model using the pre-trained embedding\n",
    "sequence_input = Input(shape=(max_len,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "x = Bidirectional(LSTM(lstm_out, recurrent_dropout=0.5, activation='tanh'))(embedded_sequences)\n",
    "preds = Dense(dense_out, activation='softmax')(x)\n",
    "\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 4465 samples, validate on 1117 samples\n",
      "Epoch 1/20\n",
      " - 85s - loss: 0.4680 - acc: 0.7783 - val_loss: 0.3523 - val_acc: 0.8496\n",
      "Epoch 2/20\n",
      " - 84s - loss: 0.2781 - acc: 0.8876 - val_loss: 0.3443 - val_acc: 0.8442\n",
      "Epoch 3/20\n",
      " - 83s - loss: 0.1968 - acc: 0.9221 - val_loss: 0.3700 - val_acc: 0.8308\n",
      "Epoch 4/20\n",
      " - 84s - loss: 0.1469 - acc: 0.9451 - val_loss: 0.4000 - val_acc: 0.8505\n",
      "Epoch 5/20\n",
      " - 82s - loss: 0.0894 - acc: 0.9711 - val_loss: 0.4802 - val_acc: 0.8478\n",
      "Epoch 6/20\n",
      " - 82s - loss: 0.0542 - acc: 0.9828 - val_loss: 0.5430 - val_acc: 0.8415\n",
      "Epoch 7/20\n",
      " - 83s - loss: 0.0479 - acc: 0.9850 - val_loss: 0.5724 - val_acc: 0.8165\n",
      "Epoch 8/20\n",
      " - 83s - loss: 0.0337 - acc: 0.9908 - val_loss: 0.6893 - val_acc: 0.8487\n",
      "Epoch 9/20\n",
      " - 83s - loss: 0.0090 - acc: 0.9989 - val_loss: 0.7429 - val_acc: 0.8397\n",
      "Epoch 10/20\n",
      " - 81s - loss: 0.0110 - acc: 0.9975 - val_loss: 0.8305 - val_acc: 0.8424\n",
      "Epoch 11/20\n",
      " - 84s - loss: 0.0045 - acc: 0.9989 - val_loss: 0.8527 - val_acc: 0.8353\n",
      "Epoch 12/20\n",
      " - 82s - loss: 0.0079 - acc: 0.9975 - val_loss: 0.8297 - val_acc: 0.8362\n",
      "Epoch 13/20\n",
      " - 81s - loss: 0.0147 - acc: 0.9942 - val_loss: 1.0953 - val_acc: 0.8236\n",
      "Epoch 14/20\n",
      " - 81s - loss: 0.0028 - acc: 0.9993 - val_loss: 1.0002 - val_acc: 0.8380\n",
      "Epoch 15/20\n",
      " - 82s - loss: 0.0011 - acc: 0.9998 - val_loss: 1.0492 - val_acc: 0.8397\n",
      "Epoch 16/20\n",
      " - 82s - loss: 2.7383e-04 - acc: 1.0000 - val_loss: 1.0793 - val_acc: 0.8469\n",
      "Epoch 17/20\n",
      " - 83s - loss: 1.4529e-04 - acc: 1.0000 - val_loss: 1.1192 - val_acc: 0.8460\n",
      "Epoch 18/20\n",
      " - 81s - loss: 1.0854e-04 - acc: 1.0000 - val_loss: 1.1464 - val_acc: 0.8478\n",
      "Epoch 19/20\n",
      " - 82s - loss: 8.5897e-05 - acc: 1.0000 - val_loss: 1.1656 - val_acc: 0.8487\n",
      "Epoch 20/20\n",
      " - 82s - loss: 7.0487e-05 - acc: 1.0000 - val_loss: 1.1859 - val_acc: 0.8487\n"
     ]
    }
   ],
   "source": [
    "model_hist_embedding = model.fit(X_train, Y_train_, epochs = 20, batch_size=batch_size, verbose = 2,\n",
    "                        validation_data=(X_test,Y_test_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XmYXHWZ9vHv03uWzt5ZOxsaQgKBEHpQg7JFIWSUKKCQVwUCmpGLbRjHd9BRdBh13lHHmWFkHCMygCIBwkBwjEFENgU0CwGykhASUtm7s3TSSe/P+8c5XalUqrsry6nTy/25rrrqLL869fRJ5dx1fmcpc3dEREQA8uIuQEREOg6FgoiIJCkUREQkSaEgIiJJCgUREUlSKIiISJJCQTo9M8s3swNmNupkthXpjhQKknPhRrnl0Wxmh1LGP3usy3P3Jnfv7e7vncy2x8vMvmBmbmZXRPUeIlExXbwmcTKzjcAX3P13bbQpcPfG3FV1YszsZWAi8Ad3n5nj985396Zcvqd0LdpTkA7HzL5tZo+a2SNmth/4nJl9yMxeM7O9ZrbNzO4xs8KwfUH4zXxMOP6LcP5vzGy/mb1qZmOPtW04/zIze9vM9pnZf5jZH83s+jZqPwU4D/gr4DIzK0ubf4WZLTezajNbb2aXhNMHmtkD4d+2x8yeCKd/wcxeSHl9pvrvNbNFZlYDfMTMLg/fY7+ZvWdm30ir4fxwXe4zs81m9vlw/W41s7yUdleb2ZJj+KeTLkChIB3Vp4BfAn2BR4FG4HZgEMFGdzrBhrc1/wf4BjAAeA/4x2Nta2aDgceAr4Tv+y5wbjt1Xwe85u7zgXeAWS0zzGwqcD/wZaAfcBGwKZz9S6CIYA9jCPDv7bxPev3/AJQCrwIHgM8RrLtPALeb2cfDGsYCvwZ+CAwEzgbecvdXgf3AtJTlfg74+THUIV2AQkE6qj+4+6/cvdndD7n7Ynf/k7s3uvsGYC5wQRuvn+/uS9y9AXgYmHwcbT8OLHf3BeG8fwUqW1uImRnweYINPOHzdSlNbgR+6u7PhX/XZndfa2YjCTbGN7n7Hnevd/eX2qg33ZPu/mq4zDp3/727rwjH3wDmcXhdfQ5Y5O6Pheuy0t2Xh/MeCudjZoPCmh45hjqkC1AoSEe1OXXEzE4zs1+b2XYzqwbuJvj23prtKcMHgd7H0XZ4ah0eHIBLtLGc84GRBHsXEITCFDM7IxwfSbD3kG4kUOnu+9pYdlvS19WHzOwFM9tlZvuAL3B4XbVWAwR7BZ80s57ANcDz7r7zOGuSTkqhIB1V+hkQPwFWAO939z7AXYBFXMM2oLxlJNwTGNFG++sI/k+9aWbbgT8S/B3XhvM3A+/L8LrNwCAz65NhXg3QM2V8aIY26etqHvAEMNLd+wL3cXhdtVYD4RlZS4CZBHs86jrqhhQK0lmUAvuAGjObQNvHE06W/yX4pv8JMysgOKZRlqlh+O36KoIuoskpjzsIDpTnAz8DvmBmF5lZnpmVm9l4d98M/A6418z6mVmhmZ0fLvoN4Ewzm2RmPYBvZlF3KbDb3WvN7IME3/pb/AKYbmZXhgetB5nZWSnzHwK+CpwGLMjivaSLUShIZ/Flgm/i+wn2Gh6N+g3dfQdwNcFB2SqCb9ivA3UZml8R1vYLd9/e8gB+CvQAPuburwBfBO4hCLjnCbpzIOzLB94GdgC3hjWsAr4LvACsBbI51nAT8E/hmVtf43B3Fu7+LsHB578DdgPLgEkpr30COIXgOMuhLN5LuhhdpyCSpfDb/lbgKnd/Oe56ohB2kb0LXO/uL8RcjsRAewoibTCz6WbW18yKCU5bbQT+HHNZUfoMwZ7Qi3EXIvEoiLsAkQ7uwwSnqRYBK4FPunum7qNOz8z+AIwDPuvqQui21H0kIiJJ6j4SEZGkTtd9NGjQIB8zZkzcZYiIdCpLly6tdPeMp1SniiwUzOx+gtsE7HT3MzLMN4L7u8wguIr0endf1t5yx4wZw5IlukeXiMixMLNN7beKtvvoAYKblrXmMoKDWuOAOcCPI6xFRESyENmegru/1HJ731bMBB4Kz3J4LbySc5i7b4uqJune9tc2UJCXR3FBHnl5Ud8h47CmZqeusYmGRqewwCguyCc/h+9/ohqbmtm2r5bt1bXUNzbT7E5Ts4fPwd/n7jSlTW9uGXYPh6HZHQ+f4fC4pwxD8FpPm49OimHahCGcNbJfpO8R5zGFERx5I69EOO2oUDCzOQR7E4wapV9RlLa5O1v2HmLFlmpWbt3Hii37WLG1ml37D59JWpQfhENxYR7FBfmHnwvyKDliOD/ZrjA/j4amZmobmqlrbKauoYnatOf6xmZqG5qC+eFwY/PRG7OCPEsuu+W56Ij3y6ckfG6pqaQgn/69ihgQPgb2KqJ/+NynpPC4g65lo5/Yc4jEnoPh8+Hh7dW1NGX4G3LNOk+ORmZwn5IuHQqZ/okzfvLcfS7BrZKpqKiI/9MpHUZzs7Np98Fww7+PlVuqWbF1H3sPNgCQn2eMG9yb88eVMW5IcPPTuoZmahubqGtopq6xKdzINyU34nWNzew91EBdy8Y93Og3NDZTVJCX3JAXpWy0+/YopLi0+MggKcg/HDApoZLN++871MDOtPc/WN9IbUNzxvWQn2f07xkExIBeRQzoXcSAnmF49A6eS0sK2Vld2+5G3wyG9imhvH8Pzh07gPL+PSjv34NhfXsk97LyzMjPM/LNyMsL3j/Pjp7eMh7MI9kGgzwDM8MIprds9FuG88J5FraT3IgzFBIcvu8LBHej3BpTLdIJNDY1s6GyJgiAcOO/ams1B+qCX+osys9j/NBSLjtjKKcP78sZI/py2tBSSgrzY6785KltaGJ3TT27a+qpqqlnd00dVQfq2XMwnHYgeF69tZqqmnr2HWo4ahmtbfTL+/dMbvyLCnS2encVZyg8DdxiZvOADwD7dDxBmpud7dW1bKyqYVPVQTZW1hwerqpJflMuKcxj4rA+XDFlBGcM78vpI/owbnBpl9+YlRTmM7xfD4b365FV+8amZvYcbGB3TT3VtQ0MLi3WRl/aFOUpqY8AFxLcJz5BcMvfQgB3/y9gIcHpqOsJTkmdHVUt0rE0NTtb9x5iY1UNG6sOsqkyfK6qYdPug9Q3Hu4iKcrPY9TAnowZ2JMPv38Qp4/owxnD+3JKWe9OdbA2LgX5eZSVFlNWWhx3KdJJRHn20ax25jtwc1TvL9Fyd+oam9lf28iBukYO1Dayv66BAy3jdY1HzDtQ18ieg/W8V3WQzXsO0tB0uA+7pDCP0QN6MXZQLy4+bTCjB/ZizMCejB7Ui6F9SrTxF8mhTndFs+TWgbpGXllfyYtv72LZe3upPtSQ3Ohnc0ZKQZ5RWlJA75IC+pQUMn5oKZecPpSxg3qGG/9eDC4tzukpoiLSOoWCHMHdWbN9Py++vYsX1u5k6aY9NDQ5vYryqRgzgAnDSiktDjbyvYsL6V1SEIwnpx05XFyQpzNHRDoRhYKw72ADL6/fxYtrd/Hi27vYGZ7PP2FYH2788ClccGoZ54zur4OTIt2AQqEbam523tqyjxffDkLg9ff20OzQp6SAj5xaxgXhY0ifkrhLFZEcUyh0E7UNTSxasZ0X1u7kpXWV7K6pxwzOHNGXWy56PxeMH8xZ5X0pyNfegEh3plDoBl55p5K/f3IF71bWMLBXUXJP4CPjBjGwt05VFJHDFApd2J6aer6zcDXzlyYYPbAnD8z+C84fV6YzfUSkVQqFLsjdefL1LXz716upPtTAzRe9j1svHtelbvcgItFQKHQxGytr+PpTK/jD+kqmjOrHd6+YxGlD+8Rdloh0EgqFLqKhqZm5L23gnufWUZSfxz9+8gw+e+4odRWJyDFRKHQBSzft4Wv/8xZrd+znsjOG8q3LT9fppCJyXBQKnVh1bQPfW7SGh//0HsP6lHDftRV8dOKQuMsSkU5ModAJuTuLVmznm0+vpPJAHddPHcOXLxlP72L9c4rIidFWpJPZsvcQ31ywgt+t3snEYX2477oKziyP9uf5RKT7UCh0Eo1NzTz46ib+5bdrcYevzTiNG84bqyuQReSkUih0Ai+s3cl3F67m7R0HuHB8Gf848wxGDugZd1ki0gUpFDqwNdur+c6vV/PyukpGD+zJjz87helnDNWtqEUkMgqFDmjn/lp++Nu3eWzJZnoXF/D1v5zA5z80muICXZEsItFSKHQgh+qbuO/lDfz4xXeob2zmuqljuO3icfTvVRR3aSLSTSgUOoDm5uBeRT/47Vq27avl0tOHcOdlExg7qFfcpYlIN6NQiNmr71TxnYWrWLGlmjPL+/JvV0/mA6cMjLssEemmFAox2bDrAP/0mzU8u2oHw/uW8G9XT+bys4brXkUiEiuFQo7trqnnnufW8YvXNlFSmM9XLh3PjR8eq9tai0iHoFDIkfrGZh545V3+4/frqalrZNa5o/jrj55KWal++UxEOg6FQg7srqnnSz9fyp837ubC8WV8bcYETh1SGndZIiJHUShEbN2O/dzw4GJ2VNfx79dMZubkEXGXJCLSKoVChJ5fu5Pbfvk6JUX5PPZXH2LySN24TkQ6NoVCBNyd+/+4ke/8ehUTwjuZDuvbI+6yRETapVA4yeobm/nm0yt45M+bmX76UH549Vn0LNJqFpHOQVurk2hPTT03PbyU1zbs5uaL3seXPzZe1x2ISKcS6c34zWy6ma01s/VmdmeG+aPN7Dkze9PMXjCz8ijridL6nQf41H/+kWWb9vKvV5/FVy49TYEgIp1OZKFgZvnAvcBlwERglplNTGv2A+Ahdz8TuBv4p6jqidLL63bxqf/8IwfqGnlkzgf51NmdNttEpJuLck/hXGC9u29w93pgHjAzrc1E4Llw+PkM8zu8h17dyPX/vZgR/Xrw1M3ncc7o/nGXJCJy3KIMhRHA5pTxRDgt1RvAleHwp4BSMzvqbnBmNsfMlpjZkl27dkVS7LFqaGrmG0+t4K4FK7lo/GDm3zSV8v76NTQR6dyiDIVMHeqeNv63wAVm9jpwAbAFaDzqRe5z3b3C3SvKyspOfqXHaN/BBmb/92J+/tom/uqCU/jJ58+hd7GO2YtI5xflliwBjEwZLwe2pjZw963AFQBm1hu40t33RVjTCXu3soYbH1jM5j0H+f5VZ/LpipHtv0hEpJOIMhQWA+PMbCzBHsA1wP9JbWBmg4Dd7t4MfBW4P8J6Ttgr6yu56eFl5OcZv/ziB/mLMQPiLklE5KSKrPvI3RuBW4BngNXAY+6+0szuNrPLw2YXAmvN7G1gCPCdqOo5US+v28W19/+ZIX2KWXDzeQoEEemSIu0Id/eFwMK0aXelDM8H5kdZw8ky96UNDC4t5ombplJaUhh3OSIikYj04rWuYtu+Q/xhfSVXnVOuQBCRLk2hkIX/WbYFd7jyHF2UJiJdm0KhHe7O/KUJzh07gNEDe8VdjohIpBQK7Vi6aQ/vVtZwlfYSRKQbUCi04/ElCXoW5fOXk4bFXYqISOQUCm04WN/Ir9/axoxJw+ilK5ZFpBtQKLRh0YrtHKhrVNeRiHQbCoU2PL4kwagBPfnAWF2oJiLdg0KhFZt3H+TVDVVcdU45ZvqxHBHpHhQKrXhiWQIzuGJK+t2+RUS6LoVCBs3NzhPLEkx930D9RoKIdCsKhQz+9O5uNu8+xKfP0W2xRaR7UShk8PjSzZQWF3Dp6UPjLkVEJKcUCmkO1DXym7e28/GzhtGjKD/uckREckqhkGbhm9s41NDEVeo6EpFuSKGQZv7SBKeU9WLKqH5xlyIiknMKhRQbK2v488bdujZBRLothUKK+UsT5BlccbZuayEi3ZNCIdQUXpvwkXFlDO1bEnc5IiKxUCiEXnmnkm37avl0hfYSRKT7UiiE5i9N0KekgI9OGBJ3KSIisVEoAPsONbBoxXZmTh5BSaGuTRCR7kuhAPzvm1upa2xW15GIdHsKBYKuo1OH9GbSiL5xlyIiEqtuHwrrd+7n9ff28ulzRuraBBHp9rp9KDy+NEF+nvHJs/W7CSIi3ToUGpuaeXLZFi4aX0ZZaXHc5YiIxK5bh8LL6yrZub9ON78TEQl161CYvzTBgF5FXHza4LhLERHpELptKOw9WM+zq3Ywc/Jwigq67WoQETlCpFtDM5tuZmvNbL2Z3Zlh/igze97MXjezN81sRpT1pFqwfCv1Tc36yU0RkRSRhYKZ5QP3ApcBE4FZZjYxrdnXgcfc/WzgGuA/o6on3fylCSYO68PE4X1y9ZYiIh1elHsK5wLr3X2Du9cD84CZaW0caNkq9wW2RlhP0prt1by1ZZ+uYBYRSRNlKIwANqeMJ8Jpqb4FfM7MEsBC4NZMCzKzOWa2xMyW7Nq164QLm78kQWG+MXOyrk0QEUkVZShkujzY08ZnAQ+4ezkwA/i5mR1Vk7vPdfcKd68oKys7oaIampp5avkWpp02hAG9ik5oWSIiXU2UoZAAUo/ilnN099CNwGMA7v4qUAIMirAmnl+zk8oD9eo6EhHJIMpQWAyMM7OxZlZEcCD56bQ27wHTAMxsAkEonHj/UBvmL00wqHcxF5x6YnscIiJdUWSh4O6NwC3AM8BqgrOMVprZ3WZ2edjsy8AXzewN4BHgendP72I6aSoP1PH7NTu5YsoICvJ1bYKISLqCKBfu7gsJDiCnTrsrZXgVcF6UNaR66vUtNDY7nz5HXUciIpl0m6/L7s78pQnOGtmPcUNK4y5HRKRDajcUzOwWM+ufi2KitHJrNWu27+cq7SWIiLQqmz2FocBiM3ssvG1Fp/wlmudW76SoII/LzxwedykiIh2WZXNcNwyCS4DZQAXBaaQ/c/d3oi3vaBUVFb5kyZJjfp27897ug4we2CuCqkREOjYzW+ruFe21y+qYQnhG0Pbw0Qj0B+ab2fdOqMocMjMFgohIO9o9+8jMbgOuAyqB+4CvuHtDeOXxOuD/RluiiIjkSjanpA4CrnD3TakT3b3ZzD4eTVkiIhKHbLqPFgK7W0bMrNTMPgDg7qujKkxERHIvm1D4MXAgZbwmnCYiIl1MNqFgqbeecPdmIr4SWkRE4pFNKGwws9vMrDB83A5siLowERHJvWxC4UvAVGALwe2wPwDMibIoERGJR7vdQO6+k+C21yIi0sVlc51CCcGP4ZxO8HsHALj7DRHWJSIiMcim++jnBPc/uhR4keAX1PZHWZSIiMQjm1B4v7t/A6hx9weBvwQmRVuWiIjEIZtQaAif95rZGUBfYExkFYmISGyyud5gbvh7Cl8n+I3l3sA3Iq1KRERi0WYohDe9q3b3PcBLwCk5qUpERGLRZvdRePXyLTmqRUREYpbNMYVnzexvzWykmQ1oeURemYiI5Fw2xxRarke4OWWao64kEZEuJ5srmsfmohAREYlfNlc0X5tpurs/dPLLERGROGXTffQXKcMlwDRgGaBQEBHpYrLpPro1ddzM+hLc+kJERLqYbM4+SncQGHeyCxERkfhlc0zhVwRnG0EQIhOBx6IsSkRE4pHNMYUfpAw3ApvcPRFRPSIiEqNsQuE9YJu71wKYWQ8zG+PuGyOtTEREci6bYwqPA80p403htHaZ2XQzW2tm683szgzz/9XMloePt81sb3Zli4hIFLLZUyhw9/qWEXevN7Oi9l5kZvnAvcDHCH7bebGZPe3uq1KWdUdK+1uBs4+leBERObmy2VPYZWaXt4yY2UygMovXnQusd/cNYajMA2a20X4W8EgWyxURkYhks6fwJeBhM/tROJ4AMl7lnGYEsDllPAF8IFNDMxsNjAV+38r8OcAcgFGjRmXx1iIicjyyuXjtHeCDZtYbMHfP9veZLdPiWml7DTDf3ZtaqWEuMBegoqKitWWIiMgJarf7yMy+a2b93P2Au+83s/5m9u0slp0ARqaMlwNbW2l7Deo6EhGJXTbHFC5z9+RZQeGvsM3I4nWLgXFmNjY8MH0Nwc95HsHMxgP9gVezK1lERKKSTSjkm1lxy4iZ9QCK22gPgLs3Evxq2zPAauAxd19pZnenHrgmOMA8z93VLSQiErNsDjT/AnjOzP47HJ8NPJjNwt19IbAwbdpdaePfymZZIiISvWwONH/PzN4EPkpw8HgRMDrqwkREJPeyvUvqdoKrmq8k+D2F1ZFVJCIisWl1T8HMTiU4ODwLqAIeJTgl9aIc1SYiIjnWVvfRGuBl4BPuvh7AzO5oo72IiHRybXUfXUnQbfS8mf3UzKaR+YI0ERHpIloNBXd/0t2vBk4DXgDuAIaY2Y/N7JIc1SciIjnU7oFmd69x94fd/eMEVyUvB466DbaIiHR+x/Qbze6+291/4u4XR1WQiIjE55hCQUREujaFgoiIJCkUREQkSaEgIiJJCgUREUlSKIiISJJCQUREkhQKIiKSpFAQEZEkhYKIiCQpFEREJEmhICIiSQoFERFJUiiIiEiSQkFERJIUCiIikqRQEBGRJIWCiIgkKRRERCRJoSAiIkkKBRERSYo0FMxsupmtNbP1ZnZnK20+Y2arzGylmf0yynpERKRtBVEt2MzygXuBjwEJYLGZPe3uq1LajAO+Cpzn7nvMbHBU9YiISPui3FM4F1jv7hvcvR6YB8xMa/NF4F533wPg7jsjrEdERNoRZSiMADanjCfCaalOBU41sz+a2WtmNj3CekREpB2RdR8BlmGaZ3j/ccCFQDnwspmd4e57j1iQ2RxgDsCoUaNOfqUiIgJEu6eQAEamjJcDWzO0WeDuDe7+LrCWICSO4O5z3b3C3SvKysoiK1hEpLuLMhQWA+PMbKyZFQHXAE+ntXkKuAjAzAYRdCdtiLAmERFpQ2Sh4O6NwC3AM8Bq4DF3X2lmd5vZ5WGzZ4AqM1sFPA98xd2roqpJRETaZu7p3fwdW0VFhS9ZsiTuMkREOhUzW+ruFe210xXNIiKSpFAQEZEkhYKIiCQpFEREJEmhICIiSQoFERFJUiiIiEiSQkFERJIUCiIikqRQEBGRJIWCiIgkKRRERCRJoSAiIkkKBRERSVIoiIhIkkJBRESSFAoiIpKkUBARkSSFgoiIJCkUREQkSaEgIiJJCgUREUlSKIiISJJCQUREkhQKIiKSpFAQEZEkhYKIiCQpFEREJEmhICIiSQoFERFJKohy4WY2Hfh3IB+4z93/X9r864HvA1vCST9y9/uirElEuo+GhgYSiQS1tbVxl5IzJSUllJeXU1hYeFyvjywUzCwfuBf4GJAAFpvZ0+6+Kq3po+5+S1R1iEj3lUgkKC0tZcyYMZhZ3OVEzt2pqqoikUgwduzY41pGlN1H5wLr3X2Du9cD84CZEb6fiMgRamtrGThwYLcIBAAzY+DAgSe0ZxRlKIwANqeMJ8Jp6a40szfNbL6Zjcy0IDObY2ZLzGzJrl27oqhVRLqo7hIILU70740yFDJV5mnjvwLGuPuZwO+ABzMtyN3nunuFu1eUlZWd5DJFRKRFlKGQAFK/+ZcDW1MbuHuVu9eFoz8FzomwHhGRnOrdu/dR09auXcuFF17I5MmTmTBhAnPmzOGZZ55h8uTJTJ48md69ezN+/HgmT57MtddeywsvvICZ8bOf/Sy5jNdffx0z4wc/+MFJrznKUFgMjDOzsWZWBFwDPJ3awMyGpYxeDqyOsB4Rkdjddttt3HHHHSxfvpzVq1dz6623cumll7J8+XKWL19ORUUFDz/8MMuXL+ehhx4CYNKkSTz66KPJZcybN4+zzjorkvoiO/vI3RvN7BbgGYJTUu9395VmdjewxN2fBm4zs8uBRmA3cH1U9YhI9/YPv1rJqq3VJ3WZE4f34ZufOP2YXrNt2zbKy8uT45MmTWr3NaNGjaK6upodO3YwePBgFi1axIwZM4653mxEep2Cuy8EFqZNuytl+KvAV6OsQUSkI7njjju4+OKLmTp1KpdccgmzZ8+mX79+7b7uqquu4vHHH+fss89mypQpFBcXR1JfpKEgItJRHOs3+qjMnj2bSy+9lEWLFrFgwQJ+8pOf8MYbb7S7kf/MZz7D1VdfzZo1a5g1axavvPJKJPXpNhciIjk2fPhwbrjhBhYsWEBBQQErVqxo9zVDhw6lsLCQZ599lmnTpkVWm/YURERyaNGiRUybNo3CwkK2b99OVVUVI0ZkuoTraHfffTc7d+4kPz8/svoUCiIiETl48OARB5X/5m/+hkQiwe23305JSQkA3//+9xk6dGhWy5s6dWokdaYy9/TryTq2iooKX7JkSdxliEgnsHr1aiZMmBB3GTmX6e82s6XuXtHea3VMQUREkhQKIiKSpFAQkS6ts3WRn6gT/XsVCiLSZZWUlFBVVdVtgqHl9xRaDmIfD519JCJdVnl5OYlEgu50y/2WX147XgoFEemyCgsLj/sXyLordR+JiEiSQkFERJIUCiIiktTprmg2s13AprjraMUgoDLuItqg+k5MR68POn6Nqu/EnEh9o9293d8z7nSh0JGZ2ZJsLiOPi+o7MR29Puj4Naq+E5OL+tR9JCIiSQoFERFJUiicXHPjLqAdqu/EdPT6oOPXqPpOTOT16ZiCiIgkaU9BRESSFAoiIpKkUDhGZjbSzJ43s9VmttLMbs/Q5kIz22dmy8PHXTmucaOZvRW+91E/U2eBe8xsvZm9aWZTcljb+JT1stzMqs3sr9Pa5Hz9mdn9ZrbTzFakTBtgZs+a2brwuX8rr70ubLPOzK7LUW3fN7M14b/fk2bWr5XXtvlZiLjGb5nZlpR/xxmtvHa6ma0NP4935rC+R1Nq22hmy1t5baTrsLVtSmyfP3fX4xgewDBgSjhcCrwNTExrcyHwvzHWuBEY1Mb8GcBvAAM+CPwppjrzge0EF9XEuv6A84EpwIqUad8D7gyH7wT+OcPrBgAbwuf+4XD/HNR2CVAQDv9zptqy+SxEXOO3gL/N4jPwDnAKUAS8kf7/Kar60ub/C3BXHOuwtW1KXJ8/7SkcI3ff5u7LwuH9wGpgRLxVHbOZwEMeeA3oZ2bDYqhjGvCOu8d+hbq7vwTsTps8E3gwHH4Q+GSGl14KPOvuu919D/AsMD3q2tz9t+7eGI6+Bhz/vZJPglbWXzbOBda7+wZ3rwfmEaz3k6qt+szMgM8Aj5zs983t8kA4AAAEOklEQVRGG9uUWD5/CoUTYGZjgLOBP2WY/SEze8PMfmNmp+e0MHDgt2a21MzmZJg/AticMp4gnmC7htb/I8a5/loMcfdtEPzHBQZnaNMR1uUNBHt+mbT3WYjaLWEX1/2tdH90hPX3EWCHu69rZX7O1mHaNiWWz59C4TiZWW/gCeCv3b06bfYygi6Rs4D/AJ7KcXnnufsU4DLgZjM7P22+ZXhNTs9NNrMi4HLg8Qyz415/xyLWdWlmfw80Ag+30qS9z0KUfgy8D5gMbCPookkX+2cRmEXbewk5WYftbFNafVmGaSe0/hQKx8HMCgn+8R529/9Jn+/u1e5+IBxeCBSa2aBc1efuW8PnncCTBLvoqRLAyJTxcmBrbqpLugxY5u470mfEvf5S7GjpVgufd2ZoE9u6DA8qfhz4rIcdzOmy+CxExt13uHuTuzcDP23lvWP9LJpZAXAF8GhrbXKxDlvZpsTy+VMoHKOw//FnwGp3/2ErbYaG7TCzcwnWc1WO6utlZqUtwwQHJFekNXsauDY8C+mDwL6W3dQcavXbWZzrL83TQMvZHNcBCzK0eQa4xMz6h90jl4TTImVm04G/Ay5394OttMnmsxBljanHqT7VynsvBsaZ2dhw7/EagvWeKx8F1rh7ItPMXKzDNrYp8Xz+ojqi3lUfwIcJds/eBJaHjxnAl4AvhW1uAVYSnEnxGjA1h/WdEr7vG2ENfx9OT63PgHsJzvp4C6jI8TrsSbCR75syLdb1RxBQ24AGgm9fNwIDgeeAdeHzgLBtBXBfymtvANaHj9k5qm09QV9yy2fwv8K2w4GFbX0Wcrj+fh5+vt4k2MANS68xHJ9BcMbNO1HVmKm+cPoDLZ+7lLY5XYdtbFNi+fzpNhciIpKk7iMREUlSKIiISJJCQUREkhQKIiKSpFAQEZEkhYJIGjNrsiPv5HrS7txpZmNS79Qp0tEUxF2ASAd0yN0nx12ESBy0pyCSpfC++v9sZn8OH+8Pp482s+fCG789Z2ajwulDLPitgzfCx9RwUflm9tPw3vm/NbMesf1RImkUCiJH65HWfXR1yrxqdz8X+BHwb+G0HxHcivxMghvT3RNOvwd40YMb+00huCIWYBxwr7ufDuwFroz47xHJmq5oFkljZgfcvXeG6RuBi919Q3gDs+3uPtDMKglu4dAQTt/m7oPMbBdQ7u51KcsYQ3D/+3Hh+N8Bhe7+7ej/MpH2aU9B5Nh4K8OttcmkLmW4CR3bkw5EoSBybK5OeX41HH6F4O6eAJ8F/hAOPwfcBGBm+WbWJ1dFihwvfUMROVoPO/JH3Be5e8tpqcVm9ieCL1Szwmm3Afeb2VeAXcDscPrtwFwzu5Fgj+Amgjt1inRYOqYgkqXwmEKFu1fGXYtIVNR9JCIiSdpTEBGRJO0piIhIkkJBRESSFAoiIpKkUBARkSSFgoiIJP1/vOBtnRpdo2UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x268b73f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Training Accuracy\n",
    "x = np.arange(20)+1\n",
    "\n",
    "plt.plot(x, model_hist_embedding.history['acc'])\n",
    "plt.legend(['LSTM'], loc='lower right')\n",
    "plt.ylabel(\"Accuracy\")\n",
    "axes = plt.gca()\n",
    "axes.set_ylim([0.45,1.01])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.title(\"Training Accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VfWd//HXJxtJCDshARJWAUFUxNSt1VK1CrZq7WKli2uH2qmjY5epnU47HR/t/Kq1M1N/te3Y2mn15yhof9PBDuJWqbZWfwQVZBFBBBIhEMIWCCHb5/fHOTlc4k1yA5x7E3g/H4/7yNnP5x4u933O99xzjrk7IiIiAFmZLkBERHoPhYKIiEQUCiIiElEoiIhIRKEgIiIRhYKIiEQUCnJCMLN+ZrbPzEZluhaR3kyhIBkVflG3v9rM7EBC/2ePYrkvm9nn2vvd/aC7F7n7lmNTedJ13mxmbmZXxLUOkbgpFCSjwi/qIncvAjYDlycMezjT9fXQdcDO8G9amVl2utcpxyeFgvRqZpZtZt82sw1mtsPMHjazweG4/mb2qJntNLPdZvaKmQ0xsx8B7wN+GR5x/MjM8sO9+LJw3kfN7N/M7CkzqzezP5vZ2IT1fsTM1oXL/beORx5J6pwMnA18EfiomQ3tMP5TZrbCzPaGy70oHD7czB40sxoz22Vm88PhN5vZswnzJ6v/XjN72sz2A+ea2VVmtjxcxyYz+/sONcwK38ceM9tsZp8xs/PNrMrMshKm+6yZvXxk/2LS1ykUpLf7OnAJ8AGgDGgG/jUc9wUgBxgNDAduAZrc/avAUuAL4RHHVztZ9meAbwJDga3APwGYWSkwH7gdKAa2AGd2U+d1wJ/c/XFgE3BN+wgzOx+4H7gNGAxcBFSFo+cDBpwMlAD3dbOeRJ8Dvg0MIHi/e8P3NBi4Cviamc0OazgJ+D3wQ2BY+H5WufuLQBPwwQ7LfagHdchxRKEgvd0XgTvcfYu7NxJ8cX/azIwgIIqBie7e4u5L3X1/D5a9wN1fdfdm4D+BGeHwK4Cl7v77cNw9wK7OFhLuZX8+XAbAIxzehPQF4Ofu/ry7t7n7Znd/y8zGA+cDf+3uu929yd1f6EH9j7v7K+EyD7r7c+6+Kux/FVjAoS/7zwNPuPtvw21V6+7Lw3EPEgQBZlYSzjO/B3XIcUShIL1W+MVfDiwKm3F2A68RfG6HAQ8AfwQeN7NqM/vnHrat1yR0NwBFYfcoDu3J4+5twLtdLOdDQCnwWNj/MHCWmZ0c9pcDbyeZrxzY7u71Pag5UVVij5m938z+aGa1ZrYHuJ7gCKqrGiAIhY+bWT4wF3jG3XccYU3SxykUpNfy4Ba+7wIXuvvghFe+u+8I946/4+4nAxcAn+JQs83R3P53K0FTFRAdCYzuYvrrCP4vrTKzGuCFcP3XhuOrgIlJ5qsCRphZUZJx+4HChP7SJNN0fI8LCPbwy919EPBrgqaprmrA3d8BVgCXExxRqOnoBKZQkN7u58APzKwcwMxGmNnlYffFZjYt/NLeC7QAreF824AJR7jOhcDZZnaZmeUAXwGGJJsw/EL/OMFe+YyE19eAz4e1/RL4opldYGZZZlZuZpPDL+MXgJ+Y2SAzyzOzC8JFvw6cYWanmFkh8J2uCg6PqoqAOndvNLPzCEKy3YMEJ8CvCk/eF5vZaR3Gf5tgmz2R4naS45BCQXq7u4FngT+YWT3wEjAzHDca+G+gHlgJLCLYW4bgZPS14S967u7JCt19K0Ezyr3ADoKjhjeAg0km/yTBz1Afcfea9hfBieWBBEc5LwI3Az8F9gDPcehIZC6QC6wjaM76UljDG+F7fxF4E1jSTc0eruOecDv9HYeas3D3t4Ergb8nOD9SCZySsIjHgJMIzrMke59ygjA9ZEeka+HRQg3BNRR/yXQ9cQiPaDYD17j7nzJdj2SOjhREkjCzOWGTTj7wjwQnopdluKw4zQX2KhAkJ9MFiPRSFxD8iiiHoGnqKndvymxJ8QgvVBtHcI2DnODUfCQiIhE1H4mISKTPNR8NHz7cx40bl+kyRET6lGXLlu1w9+LupostFMzsV8BHCa7YnJ5kvAE/Bi4jOIl3fXhpfpfGjRtHZWXlsS5XROS4ZmabUpkuzuajXwOzuxg/B5gUvuYBP4uxFhERSUFsoRDe2GtnF5NcCTzogZeBwWY2Mq56RESke5k80Tyaw2/oVU0n95cxs3lmVmlmlbW1tWkpTkTkRJTJULAkw5L+Ptbd73f3CnevKC7u9jyJiIgcoUyGQjXB7XzblRE8zERERDIkk6GwkOCGZWZm5wB7whuRiYhIhsT5k9RHgFnAcDOrJrh/TC6Au/+c4I6WlwHrCX6SekNctYiISGpiCwV3n9vNeAe+HNf6RUSk53SbCxERiSgUREQkolAQEZGIQkFERCIKBRERiSgUREQkolAQEZGIQkFERCIKBRERiSgUREQkolAQEZGIQkFERCIKBRERiSgUREQkolAQEZGIQkFERCIKBRERiSgUREQkolAQEZGIQkFERCIKBRERiSgUREQkolAQEZGIQkFERCIKBRERiSgUREQkolAQEZGIQkFERCIKBRERiSgUREQkEmsomNlsM1trZuvN7I4k48ea2XNmtsLMlphZWZz1iIhI12ILBTPLBu4D5gDTgLlmNq3DZPcAD7r7acCdwP+Kqx4REelenEcKZwHr3X2DuzcBjwJXdphmGvBc2P18kvEiIpJGcYbCaKAqob86HJZoOfCJsPsqYICZDeu4IDObZ2aVZlZZW1sbS7EiIhJvKFiSYd6h/2vAB83sNeCDwLtAy3tmcr/f3SvcvaK4uPjYVyoiIgDkxLjsaqA8ob8M2JI4gbtvAT4OYGZFwCfcfU+MNYmISBfiPFJYCkwys/FmlgdcAyxMnMDMhptZew3fBH4VYz0iItKN2ELB3VuAW4CngDXAAndfZWZ3mtkV4WSzgLVm9hZQAnw/rnpERKR75t6xmb93q6io8MrKykyXISLSp5jZMnev6G46XdEsIiIRhYKIiEQUCiIiElEoiIhIRKEgIiIRhYKIiEQUCiIiElEoiIhIRKEgIiIRhYKIiEQUCiIiElEoiIhIJM7nKfQqexqaaWptY3hRHmbJnv/Tu72zYz9L1m6nf14OAwtyGFiQy8D8XAYV5DKoMJeivByysvre+xKR3uWECYUFlVV8f9EaCnKzKRtSwJihhZQPLTysu3xoIUX9etcmWb1lLz9dsp5Fb2ylrYsb2mYZDAhDYmBBTvC3PTQKchlYkMuQwjzGDS9kYnERIwb061Ph+NL6HfzHSxsZN6yQM8YMYUb5YEYOyu9T70GOTkNTC20OudlGblbWMd0Jampp40BTKw3NLTQ0tQbdTa3sb2qJug80t9LU0kZLaxvNrW00t3r491B3S/uwNqe5pY2WtjaaWj2a52h94fwJXHpK6TF4x53rXd+AMTp/8nC+mz2Nql0HqNrZwOadDbzyzk72HTz86Z9D++dRPqSAsqGFQVgMKaR8aBAcowYXkJudnha3ZZt2cd/z6/nDm9sp6pfDFz84kc+ePQaAvQda2HOgmT0Hmtnb2MzeA8Hr0LBg/La9+6LhB1sO/0AW9cth/PD+jB/enwnF/ZlQXMSEsLswr/d8LNydf39hA3cvfpMhhXn88a1afvHiOwCUDOzHjPLBUUicVjaoV9UuR6axuZV12/axdls9b22r582aet6qqadmb+Nh02VnWRQQuTlZ5GQZudlZwbDsrMO6c7KNnKwsDra0Jv3Sb+lqj6sL7evMyTbywnW2d+dE684iL1z/0e7DZKdhJ+iEfp6Cu7O7oZnNOxuo2tVA1c4DbN7ZQPWuBqp2NvDu7gM0tx7aPvm5WcyaPII5p5Zy4ckjGJCfe0zqSKznT+t3cN/z63l5w06GFOZy0wfG8/lzxzGo4OjW1djcSt3+Jt6p3c+GHfvYULuft2uDv1v2HCDxYzByUH4QFMOLmFAcBMfE4iJGDS4gO41NVPWNzXz9sRUsXlXDR04dyV2fPI287CzWbN3L61W7eW3zLl6v2s3GugYg+JKYXDKAM8YMZkb5YGaOGcyE4UVHtEfZ2uZB2DYeCtvG5jbGDy9k3LD+5KRp5+B41tLaxsa6hsO++Nduq2dT3f7oqDgvJ4tJI4qYUjKAiSOKyMkyWto82GNvO3xvvaXVaUrcW2/fe29ro7nFaWlro19ONoV52RTkBX8L83IoyMumf142BXk54bBsCnIPjYuG5WXTLzub3JzgCz432/rUkWqqz1M4oUOhO61tTs3eRqp2BiGxonoPT62qYXv9QfKyszh/0nBmTy/lw9NKGFyYd8TraWtznl69jZ8uWc+K6j2UDsznry6YwNyzytOy59vY3MrGuv1sqN3PhjAo3t4RdNc3HjqSGpifw99ePJlrzx0b+5fiW9vqufmhZWza2cA355zMTR8Y3+l/wJ37m1gehsRrVbt5vWp3VPeA/JzgaKJ8MCePHEhTS9uhI6rDjq6a2XOgJTrqqu9wBJkoLzuLiSOKmFJSxOTSAZxcOoDJJQMYPbigT31JxME9+GJu3xM/tFfewt7GFt6u3cfamnrW1tSzvnYfTeERbJbBuGH9mRJuyymlwWvs0EIF8DGiUIhJW5vzWtUuFr1Rw+KVNby7+wA5Wca5E4cxZ/pILjmlhOFF/VJaVktrG0+s2MJPn3+bddv3MXZYIV/64ESumjmafjnZMb+T7rk7O/Y1saF2H+/s2M+ilTW88FYtU0cO5HsfO4Uzxw6NZb1PLN/CN367gsK8bH7ymZmcM2FYj+Zva3M27NjHa5t3ByGxeTdv1ux9zzmZwrzsw869DCw4/JzMoA4n83Ozs3hnx75De7U19WzZc6hJo6hfDpNLiphSOpAp7X9LBzC0/5HvMPQGLa1tvPR2HX94czt7DjTT0HR4u/uB5uBLv+FgKw3NrbR20xQzclA+k0sOhemU0gGcNKKI/NzMf+aPZwqFNHB3VlTv4cmVNSxeuZWNdQ1kGZw1fihzpo9k9vRSSgbmv2e+xuZWHl9Wzb+/8DZVOw8wpWQAf/2hiXzk1JG9eq/I3XlqVQ3/9MRqtu5p5NMV5XxjzsnH7EuvubWNHzz5Jg/86R3OHDuE+z4zk9JB791+R6KhqYUNtfujIBiQn0teztFv672NzVGzR/se8Npt9exuaI6mKR7QjyklA5hQ3J8ss7CJI/mJyuYkzR/t3WZw9vhhfHjaCM6fVEz/GH8U0dbmLNu8i4Wvb2HRG1up299EQW42w4rywqaUHPpHzSo5FOZmd2hqyQmbZIJmmMK8bPr3y2H8sP4MKjy2za6SGoVCmrk7a7bWs3jlVp5cWcO67fsAOHPsEOZML2X29FKGFObxn69s5hcvbmB7/UFmlA/mlg+dxIUnj+hTPyfdf7CFe59bxwN/eoei/Bz+7tKTueZ95Uf1HrbXN3LLw6/x/zbu5PrzxvH3l009Jl/ameDu1NYffE9QvLNjP1lmh538zM3OCk+UHt4dtFkffqK0sbmVP6+vY8+BZvKyszh34jAunlbCxVNHMHJQwTGpe9WWvTyxfAu/X7GVd3cfoF9OFhdPLeHy00cxa0qx9ub7MIVChq3fXs+Tb9Tw5MoaVm/dC0BBbjYHmlt5/0nD+PKskzh34rA+3Qb91rZ6vv27lbzyzk5mlA/mex+bzvTRg3q8nMqNO/nrh19lb2MzP/j4aXzsjNExVHt8aGlto3LTLp5dvY1n1mxjU3iS/ZRRA7l4agkfnlbCKaMG9uhztaF2HwuXb2Hh8i1sqN1PTpZxweRirjh9FBdPK+l1P9OWI6NQ6EU21e3nyZU1bKrbz9UV5ZwxZkimSzpm3J3fvf4u3/+fNezc38TnzxnLVy6ZktKvpdydX7+0ke//zxrKhhTws8+dydSRA9NQ9fHB3Xm7dh/PrtnOs6u3sWzzLtyhdGA+F00dwcXTSjh3wrCke/dbdh/giTAIVm3ZGzZNDeWK00czZ3opQ/r4eRB5L4WCpNWeA838y9NreejlTQzt349vfeRkPjZjdKd7rA1NLdzx2zdYuHwLF08t4UdXn37UP7s90dXtO8gf3tzOc2u288K6WhqaWinMy+b8ScO5eGoJZ44dwp/X72Dh8i0s3bgLgNPLB3P5aSP56Gmjjtn5G+mdFAqSESvf3cO3freS5VW7OXv8UL73selMKhlw2DTv7NjPzQ8tY932er56yRS+9MGJfeqcSl/Q2NzKXzbU8ezqbTy3ZvthF35NLiniitNHcfnpoxg7rH8Gq5R0UihIxrS1OY8ureKuxW+y/2ALN50/nlsvnET/fjk8vaqGry5YTk62ce/cMzh/UnGmyz3utZ9Afm3zLt43fignl6qJ7kSkUJCMq9t3kLsWv8mCympGDcrn/EnFzK+s4rSyQfz0szMpG1KY6RJFThiphkLf/M2f9AnDivpx9ydP5/Gbz2VgQS7zK6uYe9YYFnzxXAWCSC+l35pJ7CrGDeX3f/MBNuzYz+QO5xdEpHfRkYKkRU52lgJBpA9QKIiISCTWUDCz2Wa21szWm9kdScaPMbPnzew1M1thZpfFWY+IiHQttlAws2zgPmAOMA2Ya2bTOkz2D8ACdz8DuAb4aVz1iIhI9+I8UjgLWO/uG9y9CXgUuLLDNA60/2h6ELAlxnpERKQbcYbCaKAqob86HJbou8DnzKwaWAT8TbIFmdk8M6s0s8ra2to4ahUREeINhWT3Leh4pdxc4NfuXgZcBjxkZu+pyd3vd/cKd68oLtYVsCIicYkzFKqB8oT+Mt7bPHQTsADA3f8C5APDY6xJRES6EGcoLAUmmdl4M8sjOJG8sMM0m4GLAMxsKkEoqH1IRCRDYgsFd28BbgGeAtYQ/MpolZndaWZXhJN9FfgrM1sOPAJc733tZkwiIseRWG9z4e6LCE4gJw77TkL3auD9cdYgIiKp0xXNIiIS6TYUzOwWMzt+nh8pIiKdSuVIoRRYamYLwttW6BFZIiLHqW5Dwd3/AZgEPABcD6wzs382s4kx1yYiImmW0jmF8BdBNeGrBRgCPG5md8dYm4iIpFm3vz4ys1uB64AdwC+Br7t7c3jl8Trg7+ItUURE0iWVn6QOBz7u7psSB7p7m5l9NJ6yREQkE1JpPloE7GzvMbMBZnY2gLuviaswERFJv1RC4WfAvoT+/eEwERE5zqQSCpZ46wl3byPmK6FFRCQzUgmFDWZ2q5nlhq/bgA1xFyYiIumXSijcDJwHvEtwO+yzgXlxFiUiIpnRbTOQu28nuO21iEif0tzcTHV1NY2NjZkuJW3y8/MpKysjNzf3iOZP5TqFfIKH4ZxC8LwDANz9xiNao4hImlRXVzNgwADGjRvHiXCHHnenrq6O6upqxo8ff0TLSKX56CGC+x9dCvyR4Alq9Ue0NhGRNGpsbGTYsGEnRCAAmBnDhg07qiOjVELhJHf/NrDf3X8DfAQ49YjXKCKSRidKILQ72vebSig0h393m9l0YBAw7qjWKiIivVIqoXB/+DyFfyB4xvJq4K5YqxIROQ4UFRW9Z9jatWuZNWsWM2bMYOrUqcybN4+nnnqKGTNmMGPGDIqKipgyZQozZszg2muvZcmSJZgZDzzwQLSM1157DTPjnnvuOeY1d3miObzp3V533wW8AEw45hWIiJxAbr31Vm6//XauvPJKAN544w1OPfVULr30UgBmzZrFPffcQ0VFBQBLlizh1FNPZf78+dx0000APProo5x++umx1NflkUJ49fItsaxZROQEtHXrVsrKyqL+U0/t/hTtmDFjaGxsZNu2bbg7ixcvZs6cObHUl8rtKp4xs68B8wnuewSAu+/sfBYRkd7ln55Yxeote4/pMqeNGsg/Xn5Kj+a5/fbbufDCCznvvPO45JJLuOGGGxg8eHC3833yk5/kscce44wzzmDmzJn069fvSMvuUirnFG4EvkzQfLQsfFXGUo2IyHHuhhtuYM2aNXzqU59iyZIlnHPOORw8eLDb+a6++moee+wxHnnkEebOnRtbfalc0XxkV0CIiPQiPd2jj9OoUaO48cYbufHGG5k+fTorV67kzDPP7HKe0tJScnNzeeaZZ/jxj3/MSy+9FEttqVzRfG2y4e7+4LEvR0Tk+LZ48WIuuugicnNzqampoa6ujtGjR6c075133sn27dvJzs6Orb5Uzim8L6E7H7gIeBVQKIiIdKGhoeGwk8pf+cpXqK6u5rbbbiM/P7hr0A9/+ENKS0tTWt55550XS52JLOFRCanNYDYIeMjdr4inpK5VVFR4ZaVOaYhI99asWcPUqVMzXUbaJXvfZrbM3Su6mzeVE80dNQCTjmA+ERHp5VI5p/AE0H44kQVMAxbEWZSIiGRGKucUEq+jbgE2uXt1TPWIiBxT7n5C3RSvp6cEOkolFDYDW929EcDMCsxsnLtvPKo1i4jELD8/n7q6uhPm9tntz1NoP4l9JFIJhccIHsfZrjUc9r7kkx9iZrOBHwPZwC/d/Qcdxv8r8KGwtxAY4e7dX9onIpKCsrIyqqurqa2tzXQpadP+5LUjlUoo5Lh7U3uPuzeZWV53M5lZNnAf8GGCZzsvNbOF7r46YVm3J0z/N8AZPSleRKQrubm5R/wEshNVKr8+qjWz6OenZnYlsCOF+c4C1rv7hjBUHgWu7GL6ucAjKSxXRERiksqRws3Aw2b2k7C/Gkh6lXMHo4GqhP5q4OxkE5rZWGA88IdOxs8D5kFwt0AREYlHKvc+ehs4x8yKCC52S/X5zMnO6nR2Wvwa4HF3b+2khvuB+yG4eC3F9YuISA9123xkZv9sZoPdfZ+715vZEDP7XgrLrgbKE/rLgC2dTHsNajoSEcm4VM4pzHH33e094VPYLkthvqXAJDMbH56YvobgcZ6HMbMpwBDgL6mVLCIicUklFLLNLHqag5kVAN0+3cHdWwie2vYUsAZY4O6rzOzOxBPXBCeYH/WjveJCRESOWionmv8P8JyZ/UfYfwPwm1QW7u6LgEUdhn2nQ/93U1mWiIjEL5UTzXeb2QrgYoKTx4uBsXEXJiIi6ZfqXVJrgDbgEwTPU1gTW0UiIpIxnR4pmNlkgpPDc4E6YD7BT1I/1Nk8IiLSt3XVfPQm8CJwubuvBzCz27uYXkRE+riumo8+QdBs9LyZ/cLMLiL5BWkiInKc6DQU3P2/3P3TwMnAEuB2oMTMfmZml6SpPhERSaNuTzS7+353f9jdP0pwVfLrwB2xVyYiImnXo2c0u/tOd/93d78wroJERCRzehQKIiJyfFMoiIhIRKEgIiIRhYKIiEQUCiIiElEoiIhIRKEgIiIRhYKIiEQUCiIiElEoiIhIRKEgIiIRhYKIiEQUCiIiElEoiIhIRKEgIiIRhYKIiEQUCiIiElEoiIhIRKEgIiIRhYKIiEQUCiIiEok1FMxstpmtNbP1ZnZHJ9NcbWarzWyVmf1nnPWIiEjXcuJasJllA/cBHwaqgaVmttDdVydMMwn4JvB+d99lZiPiqkdERLoX55HCWcB6d9/g7k3Ao8CVHab5K+A+d98F4O7bY6xHRES6EWcojAaqEvqrw2GJJgOTzezPZvaymc2OsR4REelGbM1HgCUZ5knWPwmYBZQBL5rZdHfffdiCzOYB8wDGjBlz7CsVEREg3iOFaqA8ob8M2JJkmv9292Z3fwdYSxASh3H3+929wt0riouLYytYROREF2coLAUmmdl4M8sDrgEWdpjmd8CHAMxsOEFz0oYYaxIRkS7EFgru3gLcAjwFrAEWuPsqM7vTzK4IJ3sKqDOz1cDzwNfdvS6umkREpGvm3rGZv3erqKjwysrKTJchItKnmNkyd6/objpd0SwiIhGFgoiIRBQKIiISUSiIiEhEoSAiIhGFgoiIRBQKIiISUSiIiEhEoSAiIhGFgoiIRBQKIiISUSiIiEhEoSAiIhGFgoiIRBQKIiISUSiIiEhEoSAiIhGFgoiIRBQKIiISUSiIiEhEoSAiIhGFgoiIRBQKIiISUSiIiEhEoSAiIhGFgoiIRBQKIiISUSiIiEhEoSAiIhGFgoiIRGINBTObbWZrzWy9md2RZPz1ZlZrZq+Hry/EWY+IiHQtJ64Fm1k2cB/wYaAaWGpmC919dYdJ57v7LXHVISIiqYvzSOEsYL27b3D3JuBR4MoY1yciIkcpzlAYDVQl9FeHwzr6hJmtMLPHzaw82YLMbJ6ZVZpZZW1tbRy1iogI8YaCJRnmHfqfAMa5+2nAs8Bvki3I3e939wp3ryguLj7GZYqISLs4Q6EaSNzzLwO2JE7g7nXufjDs/QVwZoz1iIhIN+IMhaXAJDMbb2Z5wDXAwsQJzGxkQu8VwJoY6xERkW7E9usjd28xs1uAp4Bs4FfuvsrM7gQq3X0hcKuZXQG0ADuB6+OqR0REumfuHZv5e7eKigqvrKzMdBkiIn2KmS1z94ruptMVzSIiElEoiIhIRKEgIiIRhYKIiEQUCiIiElEoiIhIRKEgIiIRhYKIiEQUCiIiElEoiIhIRKEgIiIRhYKIiEQUCiIiElEoiIhIpM/dOtvMaoFNma6jE8OBHZkuoguq7+j09vqg99eo+o7O0dQ31t27fZ5xnwuF3szMKlO5X3mmqL6j09vrg95fo+o7OumoT81HIiISUSiIiEhEoXBs3Z/pArqh+o5Ob68Pen+Nqu/oxF6fzimIiEhERwoiIhJRKIiISESh0ENmVm5mz5vZGjNbZWa3JZlmlpntMbPXw9d30lzjRjN7I1x3ZZLxZmb3mtl6M1thZjPTWNuUhO3yupntNbO/7TBN2refmf3KzLab2cqEYUPN7BkzWxf+HdLJvNeF06wzs+vSVNsPzezN8N/vv8xscCfzdvlZiLnG75rZuwn/jpd1Mu9sM1sbfh7vSGN98xNq22hmr3cyb6zbsLPvlIx9/txdrx68gJHAzLB7APAWMK3DNLOA32ewxo3A8C7GXwY8CRhwDvBKhurMBmoILqrJ6PYDLgBmAisTht0N3BF23wHclWS+ocCG8O+QsHtIGmq7BMgJu+9KVlsqn4WYa/wu8LUUPgNvAxOAPGB5x/9PcdXXYfyPgO9kYht29p2Sqc+fjhR6yN23uvurYXc9sAZOuzWiAAAEqElEQVQYndmqeuxK4EEPvAwMNrORGajjIuBtd8/4Feru/gKws8PgK4HfhN2/AT6WZNZLgWfcfae77wKeAWbHXZu7P+3uLWHvy0DZsVxnT3Wy/VJxFrDe3Te4exPwKMF2P6a6qs/MDLgaeORYrzcVXXynZOTzp1A4CmY2DjgDeCXJ6HPNbLmZPWlmp6S1MHDgaTNbZmbzkowfDVQl9FeTmWC7hs7/I2Zy+7UrcfetEPzHBUYkmaY3bMsbCY78kunusxC3W8Imrl910vzRG7bf+cA2d1/Xyfi0bcMO3ykZ+fwpFI6QmRUBvwX+1t33dhj9KkGTyOnA/wZ+l+by3u/uM4E5wJfN7IIO4y3JPGn9bbKZ5QFXAI8lGZ3p7dcTGd2WZvYtoAV4uJNJuvssxOlnwERgBrCVoImmo4x/FoG5dH2UkJZt2M13SqezJRl2VNtPoXAEzCyX4B/vYXf/vx3Hu/ted98Xdi8Ccs1seLrqc/ct4d/twH8RHKInqgbKE/rLgC3pqS4yB3jV3bd1HJHp7ZdgW3uzWvh3e5JpMrYtw5OKHwU+62EDc0cpfBZi4+7b3L3V3duAX3Sy7ox+Fs0sB/g4ML+zadKxDTv5TsnI50+h0ENh++MDwBp3/5dOpikNp8PMziLYznVpqq+/mQ1o7yY4Ibmyw2QLgWvDXyGdA+xpP0xNo073zjK5/TpYCLT/muM64L+TTPMUcImZDQmbRy4Jh8XKzGYD3wCucPeGTqZJ5bMQZ42J56mu6mTdS4FJZjY+PHq8hmC7p8vFwJvuXp1sZDq2YRffKZn5/MV1Rv14fQEfIDg8WwG8Hr4uA24Gbg6nuQVYRfBLipeB89JY34RwvcvDGr4VDk+sz4D7CH718QZQkeZtWEjwJT8oYVhGtx9BQG0Fmgn2vm4ChgHPAevCv0PDaSuAXybMeyOwPnzdkKba1hO0Jbd/Bn8eTjsKWNTVZyGN2++h8PO1guALbmTHGsP+ywh+cfN2XDUmqy8c/uv2z13CtGndhl18p2Tk86fbXIiISETNRyIiElEoiIhIRKEgIiIRhYKIiEQUCiIiElEoiHRgZq12+J1cj9mdO81sXOKdOkV6m5xMFyDSCx1w9xmZLkIkE3SkIJKi8L76d5nZ/wtfJ4XDx5rZc+GN354zszHh8BILnnWwPHydFy4q28x+Ed47/2kzK8jYmxLpQKEg8l4FHZqPPp0wbq+7nwX8BPi3cNhPCG5FfhrBjenuDYffC/zRgxv7zSS4IhZgEnCfu58C7AY+EfP7EUmZrmgW6cDM9rl7UZLhG4EL3X1DeAOzGncfZmY7CG7h0BwO3+ruw82sFihz94MJyxhHcP/7SWH/N4Bcd/9e/O9MpHs6UhDpGe+ku7NpkjmY0N2Kzu1JL6JQEOmZTyf8/UvY/RLB3T0BPgv8Kex+DvgSgJllm9nAdBUpcqS0hyLyXgV2+EPcF7t7+89S+5nZKwQ7VHPDYbcCvzKzrwO1wA3h8NuA+83sJoIjgi8R3KlTpNfSOQWRFIXnFCrcfUemaxGJi5qPREQkoiMFERGJ6EhBREQiCgUREYkoFEREJKJQEBGRiEJBREQi/x8g5hzm98x/gwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x269292828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Testing Accuracy\n",
    "x = np.arange(20)+1\n",
    "\n",
    "plt.plot(x, model_hist_embedding.history['val_acc'])\n",
    "plt.legend(['LSTM'], loc='right')\n",
    "plt.ylabel(\"Accuracy\")\n",
    "axes = plt.gca()\n",
    "axes.set_ylim([0.45,1.01])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.title(\"Testing Accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Temporal Convolutional Networks\n",
    "How does the simple bidirectional LSTM compare to the temporal convolutional network?\n",
    "\n",
    "This As defined in \"Text Understanding from Scratch\" by Zhang, LeCun 2015 https://arxiv.org/pdf/1502.01710v4.pdf this model is a series of 1D CNNs, with a maxpooling and fully connected layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "zhang_lecun_vocab=list(\"abcdefghijklmnopqrstuvwxyz0123456789-,;.!?:’’’/\\|_@#$%ˆ&*˜‘+-=<>()[]{}\") \n",
    "vocab_hash = {b: a for a, b in enumerate(zhang_lecun_vocab)}\n",
    "\n",
    "def to_one_hot(txt, length = max_len, vocab=vocab_hash):\n",
    "    \n",
    "    vocab_size = len(vocab.keys())\n",
    "    one_hot_vec = np.zeros((vocab_size, length), dtype=np.float32)\n",
    "    # run through txt and \"switch on\" relevant positions in one-hot vector\n",
    "    for idx, char in enumerate(txt):\n",
    "        try:\n",
    "            vocab_idx = vocab_hash[char]\n",
    "            one_hot_vec[vocab_idx, idx] = 1\n",
    "        # raised if character is out of vocabulary\n",
    "        except:\n",
    "            #print(char)\n",
    "            pass\n",
    "    return one_hot_vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 1., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "to_one_hot(\"a cat ate foccacia\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def one_hot_cnn(dense_out, max_len=300, frame='small'):\n",
    "\n",
    "    if frame == 'large':\n",
    "        cnn_size = 1024\n",
    "        fully_connected = [2048,2048,dense_out]\n",
    "    elif frame =='small':\n",
    "        cnn_size = 256\n",
    "        fully_connected = [1024,1024,dense_out]\n",
    "    elif frame =='tiny':\n",
    "        cnn_size = 128\n",
    "        fully_connected = [512,512,dense_out]\n",
    "        \n",
    "\n",
    "    model = Sequential()\n",
    "\n",
    "    model.add(Conv1D(cnn_size, 7, padding='same', input_shape=(67, max_len)))\n",
    "    model.add(MaxPooling1D(pool_size=3))\n",
    "\n",
    "    #Input = 22 x cnn_size\n",
    "    model.add(Conv1D(cnn_size, 7, padding='same'))\n",
    "    model.add(MaxPooling1D(pool_size=3))\n",
    "\n",
    "    #Input = 7 x cnn_size\n",
    "    model.add(Conv1D(cnn_size,3, padding='same'))\n",
    "\n",
    "    #Input = 7 x cnn_size\n",
    "    model.add(Conv1D(cnn_size,3, padding='same'))\n",
    "\n",
    "    model.add(Conv1D(cnn_size,3, padding='same'))\n",
    "\n",
    "    # #Input = 7 x cnn_size\n",
    "    model.add(Conv1D(cnn_size, 3, padding='same'))\n",
    "    model.add(MaxPooling1D(pool_size=3))\n",
    "    \n",
    "    model.add(Flatten())\n",
    "\n",
    "    #Fully Connected Layers\n",
    "\n",
    "    #Input is 512 Output is 1024/2048\n",
    "    model.add(Dense(fully_connected[0]))\n",
    "    model.add(Dropout(0.75))\n",
    "    model.add(Activation('relu'))\n",
    "\n",
    "    #Input is 1024/2048 Output is 1024/2048\n",
    "    model.add(Dense(fully_connected[1]))\n",
    "    model.add(Dropout(0.75))\n",
    "    model.add(Activation('relu'))\n",
    "\n",
    "    #Input is 1024/2048 Output is dense_out size (number of classes)\n",
    "    model.add(Dense(fully_connected[2]))\n",
    "    model.add(Activation('softmax'))\n",
    "\n",
    "    #Stochastic gradient parameters as set by paper\n",
    "    sgd = SGD(lr=0.01, decay=1e-5, momentum=0.9, nesterov=True)\n",
    "    model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4465, 67, 300)\n"
     ]
    }
   ],
   "source": [
    "X_train_a_ = [to_one_hot(x) for x in X_train_]\n",
    "X_test_a_ = [to_one_hot(x) for x in X_test_]\n",
    "\n",
    "X_train_a = np.asarray(X_train_a_)\n",
    "Y_train_a = Y_train_\n",
    "\n",
    "X_test_a = np.asarray(X_test_a_)\n",
    "Y_test_a = Y_test_\n",
    "\n",
    "print(X_train_a.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv1d_19 (Conv1D)           (None, 67, 256)           537856    \n",
      "_________________________________________________________________\n",
      "max_pooling1d_10 (MaxPooling (None, 22, 256)           0         \n",
      "_________________________________________________________________\n",
      "conv1d_20 (Conv1D)           (None, 22, 256)           459008    \n",
      "_________________________________________________________________\n",
      "max_pooling1d_11 (MaxPooling (None, 7, 256)            0         \n",
      "_________________________________________________________________\n",
      "conv1d_21 (Conv1D)           (None, 7, 256)            196864    \n",
      "_________________________________________________________________\n",
      "conv1d_22 (Conv1D)           (None, 7, 256)            196864    \n",
      "_________________________________________________________________\n",
      "conv1d_23 (Conv1D)           (None, 7, 256)            196864    \n",
      "_________________________________________________________________\n",
      "conv1d_24 (Conv1D)           (None, 7, 256)            196864    \n",
      "_________________________________________________________________\n",
      "max_pooling1d_12 (MaxPooling (None, 2, 256)            0         \n",
      "_________________________________________________________________\n",
      "flatten_4 (Flatten)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_12 (Dense)             (None, 1024)              525312    \n",
      "_________________________________________________________________\n",
      "dropout_7 (Dropout)          (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "dense_13 (Dense)             (None, 1024)              1049600   \n",
      "_________________________________________________________________\n",
      "dropout_8 (Dropout)          (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "activation_11 (Activation)   (None, 1024)              0         \n",
      "_________________________________________________________________\n",
      "dense_14 (Dense)             (None, 2)                 2050      \n",
      "_________________________________________________________________\n",
      "activation_12 (Activation)   (None, 2)                 0         \n",
      "=================================================================\n",
      "Total params: 3,361,282\n",
      "Trainable params: 3,361,282\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model_cnn = one_hot_cnn(dense_out, frame='small')\n",
    "model_cnn.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 4465 samples, validate on 1117 samples\n",
      "Epoch 1/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.7110 - acc: 0.5050 - val_loss: 0.6953 - val_acc: 0.4879\n",
      "Epoch 2/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.7048 - acc: 0.5024 - val_loss: 0.6945 - val_acc: 0.4879\n",
      "Epoch 3/20\n",
      "4465/4465 [==============================] - 11s 3ms/step - loss: 0.7010 - acc: 0.5122 - val_loss: 0.6922 - val_acc: 0.5264\n",
      "Epoch 4/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.6973 - acc: 0.5131 - val_loss: 0.6916 - val_acc: 0.5273\n",
      "Epoch 5/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.6886 - acc: 0.5442 - val_loss: 0.6920 - val_acc: 0.5228\n",
      "Epoch 6/20\n",
      "4465/4465 [==============================] - 11s 3ms/step - loss: 0.6528 - acc: 0.6206 - val_loss: 0.7426 - val_acc: 0.4960\n",
      "Epoch 7/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.4952 - acc: 0.7695 - val_loss: 0.8299 - val_acc: 0.5389\n",
      "Epoch 8/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.1523 - acc: 0.9485 - val_loss: 1.5280 - val_acc: 0.5443\n",
      "Epoch 9/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0866 - acc: 0.9702 - val_loss: 1.6476 - val_acc: 0.5282\n",
      "Epoch 10/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0571 - acc: 0.9801 - val_loss: 2.1418 - val_acc: 0.5515\n",
      "Epoch 11/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0632 - acc: 0.9747 - val_loss: 2.2884 - val_acc: 0.5407\n",
      "Epoch 12/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0462 - acc: 0.9850 - val_loss: 2.0141 - val_acc: 0.5443\n",
      "Epoch 13/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0269 - acc: 0.9917 - val_loss: 2.1857 - val_acc: 0.5551\n",
      "Epoch 14/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0273 - acc: 0.9908 - val_loss: 2.2976 - val_acc: 0.5568\n",
      "Epoch 15/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0175 - acc: 0.9944 - val_loss: 2.6683 - val_acc: 0.5586\n",
      "Epoch 16/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0254 - acc: 0.9906 - val_loss: 2.3203 - val_acc: 0.5363\n",
      "Epoch 17/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0141 - acc: 0.9946 - val_loss: 2.3924 - val_acc: 0.5604\n",
      "Epoch 18/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0048 - acc: 0.9987 - val_loss: 2.5982 - val_acc: 0.5542\n",
      "Epoch 19/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0037 - acc: 0.9989 - val_loss: 3.4331 - val_acc: 0.5264\n",
      "Epoch 20/20\n",
      "4465/4465 [==============================] - 12s 3ms/step - loss: 0.0107 - acc: 0.9964 - val_loss: 3.2068 - val_acc: 0.5425\n"
     ]
    }
   ],
   "source": [
    "model_hist_cnn = model_cnn.fit(X_train_a, Y_train_a, batch_size=64, epochs=20, verbose=1, \n",
    "          validation_data=(X_test_a,Y_test_a))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do these different performances compare?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8VPW5+PHPk8kGSdiSsIYlLLJLhBTXKoqtgq1Uq1W7uFaqt9bW9vZXq61a761tb2m9ba/trVqrtXXFa7GWYi2KG4ssiiJrQEgCBJIACUnIOs/vj3MShpBlAjlzJjPP+/Wa19m+c+bJYTjPfL/fc75HVBVjjDEGIMHvAIwxxkQPSwrGGGNaWFIwxhjTwpKCMcaYFpYUjDHGtLCkYIwxpoUlBdPjiUhARKpEZER3ljUmHllSMBHnnpSbX0ERORKy/KWu7k9Vm1Q1XVULu7PsiRKRr4qIisjlXn2GMV4Ru3nN+ElEdgJfVdV/dVAmUVUbIxfVyRGRt4BJwNuqOi/Cnx1Q1aZIfqaJLVZTMFFHRP5TRJ4VkadF5DDwZRE5U0RWisghEdkrIr8WkSS3fKL7y3yUu/xnd/s/ROSwiKwQkdyulnW3zxGRrSJSISK/EZF3ROT6DmIfDZwNfA2YIyLZrbZfLiLvi0iliBSIyKfd9Zki8rj7tx0UkRfc9V8VkWUh728r/odEZImIVAOfFJFL3c84LCKFIvLDVjGc6x7LChEpEpGvuMd3j4gkhJS7SkTWdOGfzsQASwomWl0GPAX0BZ4FGoFvAlk4J92LcU687fki8ENgAFAI/EdXy4rIQOA54Lvu534MzOwk7uuAlaq6ENgOXNO8QUTOAh4DvgP0A84HdrmbnwKScWoYg4BfdfI5reP/EZABrACqgC/jHLvPAt8Ukc+4MeQCfwd+CWQCpwEfquoK4DAwO2S/Xwae7EIcJgZYUjDR6m1V/ZuqBlX1iKquVtVVqtqoqjuAh4HzOnj/QlVdo6oNwF+AvBMo+xngfVVd5G57EChrbyciIsBXcE7wuNPrQorcBDyiqkvdv6tIVbeIyHCck/GtqnpQVetV9c0O4m3tRVVd4e6zTlVfU9UN7vJ64BmOHqsvA0tU9Tn3WJap6vvutj+52xGRLDemp7sQh4kBlhRMtCoKXRCRCSLydxEpEZFK4H6cX+/tKQmZrwHST6Ds0NA41OmAK+5gP+cCw3FqF+AkhekiMsVdHo5Te2htOFCmqhUd7LsjrY/VmSKyTERKRaQC+CpHj1V7MYBTK/iciPQGrgZeV9X9JxiT6aEsKZho1foKiN8DG4CxqtoHuAcQj2PYC+Q0L7g1gWEdlL8O5//UByJSAryD83dc624vAsa08b4iIEtE+rSxrRroHbI8uI0yrY/VM8ALwHBV7Qs8ytFj1V4MuFdkrQHm4dR4rOkoDllSMD1FBlABVIvIRDruT+guL+P80v+siCTi9Glkt1XQ/XV9BU4TUV7I6w6cjvIA8AfgqyJyvogkiEiOiIxX1SLgX8BDItJPRJJE5Fx31+uBU0Vkqoj0Au4NI+4M4ICq1orIGTi/+pv9GbhYRD7vdlpnici0kO1/Ar4PTAAWhfFZJsZYUjA9xXdwfokfxqk1POv1B6rqPuAqnE7Zcpxf2O8BdW0Uv9yN7c+qWtL8Ah4BegGfUtXlwM3Ar3ES3Os4zTngtuUDW4F9wDfcGDYCDwDLgC1AOH0NtwI/ca/cuoujzVmo6sc4nc/fAw4A64CpIe99ARiN089yJIzPMjHG7lMwJkzur/09wBWq+pbf8XjBbSL7GLheVZf5HI7xgdUUjOmAiFwsIn1FJAXnstVG4F2fw/LSF3BqQm/4HYjxR6LfARgT5c7BuUw1GfgI+JyqttV81OOJyNvAOOBLak0Iccuaj4wxxrSw5iNjjDEtelzzUVZWlo4aNcrvMIwxpkdZu3Ztmaq2eUl1KM+Sgog8hjNMwH5VndLGdsEZ32Uuzl2k16vqus72O2rUKNassTG6jDGmK0RkV+elvG0+ehxn0LL2zMHp1BoHzAd+52EsxhhjwuBZTUFV32we3rcd84A/uVc5rHTv5Byiqnu9isnEt8O1DSQmJJCSmEBCgtcjZBzVFFTqGptoaFSSEoWUxACBCH7+yWpsCrK3opaSylrqG4MEVWkKqjt1/j5VpanV+mDzvKo7D0FV1J3C0WUNmQfnvdpqO9F0UYwqoAhBREEIOsst6xXRIIBbxlnfEOhNU0LyCX/s7ImDmDa8X3f8Be3ys09hGMcO5FXsrjsuKYjIfJzaBCNG2FMUTcdUld2HjrBhdyUf7algw+4KNuyppPTw0StJkwNOckhJSiAlMXB0mphA6jHzgZZySYEEGpqC1DYEqWsMUtfQRG2raX1jkNqGJme7O98YPP5klpggLftuniYf83kBUt1pc0ypiQH6pyUzwH1lpiXT3532SU064UTXfNIvPniE4oM17vTofEllLU1t/A2RJh7l0QBNZFLJQDnIQDnEQA6RLYecZQ4xUJzlLCpIookEObljUadJVNKbw9qbw/SikjQOay8O05tKTeMwzfO9j5sOSTszppNCW//EbR5tVX0YZ6hk8vPz/f92mqgRDCq7DtS4J/4KPtpdyYY9FRyqaQAgkCCMG5jOueOyGTfIGfy0riFIbWMTdQ1B6hqb3JN8U8tJvK4xyKEjDdQ1n9zdk35DY5DkxISWE3lyyEm7b68kUjJSjk0kiYGjCSYkqYTz+RVHGtjf6vNr6hupbQi2eRwCCUL/3k6CGJCWzID0ZAb0dpNHujPNSE1if2Vtpyd9ERjcJ5Wc/r2YmTuAnP69yOnfiyF9e7XUshJECCQIARESEpzPT5BW6yVIoK6CpNpyEo+UE6g7CKn9IX0gwfSBJKT2RRISECBBpOWk3zyfIIK48ciJZISGWqja57wOl4RMS+DwvqPT6lLaPPX0GgAZgyF9BGTMhLRsSEwBSQDEDSx0Xtz5hDa2uS319VWk1FaQXVtJdl0l1FZCbQXUHYLaQqirhIaa9v+mpAU4w1J5x8+kUMzRcV/AGY1yj0+xmB6gsSnIjrJqJwG4J/+NeyqpqnOe1JkcSGD84AzmTBnM5KF9mTKsLxMGZ5CaFPA58u5T29DEgep6DlTXU15dz4HqOsqr6jlY466rcqab9lRSXl1PxZEGEgiSI6WMlH0oQiMB+qT1JrdPb/KHZJA1IY2B/fowsH8fhvRPZ2C/PiQnJ0MgGQJJkBBy/BpqnZNodSlUl0H1/qPzVSHz1aVQUwbBDp6iGkiGtIGQHvJKGwjpg0LWDXJOxikZR6sKdYdDTuolbZz03WntoeM/UxKcz8gYBBlDYehpkD7YWU4f7CaBQc4r8cSbeU5KU4OTLOoq3KmbOGorIecTnn+8n0nhJeA2EXkGOB2osP4EEwwqJZW17CyvZld5DTvLqo/Ol1e3/FJOTUpg0pA+XD59GFOG9mXysD6MG5hBcmJs33qTmhRgaL9eDO3X6/iNTQ1wYAeUbnFfm9HSzVBegDTWHlu2AWeIv/IwPlQSnBO4JLT/KzYpDdKynBN4v+Ew7DRnPvTVq59zcqva777cX+lV+6ByN+x530ky2kZtKLEX9B4ARw5BQ/Xx2wPJR0/umWNh1DmtTvbuNC3r2CQXjQJJkJbpvHzg5SWpTwOzcMaJL8YZ8jcJQFX/F1iMczlqAc4lqTd4FYuJLk1BZc+hI+wsr2ZneQ27ytxpeTW7DtRQ33j0pJAcSGBEZm9GZfbmnLFZTB7WhylD+zI6O71HddZ2q4ZaKC+A0s3Oyb/MTQLlBcf+Mu87AskeD6NnQfYEGDDaOSE2NUBTvTMNNs83utN6Zx/N20PLBJugV3/3F33zyd5NBMlp3fO3BZug5oCTHKr2HZtAag44iSV90NFf9M3TXv2963SIMz1umIv8/Hy1+xT8p6rUNQY5XNtIVV0jVbWNHK5roKp5ua7xmG1VdY0crKmnsLyGooM1NDQd/d6lJiUwckAaIzN7k5uVxsjMNEZl9mZkVhqD+6TG78kfnJN1yQew820oXAmlm+DgzqO/piUB+uc6J/3s8UdfWad034naxAQRWauq+Z2V63F3NJvIqqprZHlBGW9sLWVd4SEqjzS0nPTDuSIlMUHISE0kPTWRPqlJjB+cwacnDyY3q7d78k9jYEZKRC8RjWpNjVCy3kkCO9+BwhVOmzI4zSKDT4WpV7on/wkwYAwkpfobs4kplhTMMVSVzSWHeWNrKcu27GftroM0NClpyQHyRw1g4pAMMlKck3x6ShLpqYnOcsu6Y+dTEhNO7MqRaFFVCoXLYdcK2L/RaaroPxL6jYT+o5z5PsNOvJ36mCTwtvM59YedbVmnwNQrnPbxkec47eLGeMySgqGipoG3Ckp5Y0spb2wtZb97Pf/EIX246ZzRnHdKNjNG9o/5TlxU4dAu58TcnAjKtznbEnvBwIlw8GPY8AJo09H3JSRC3+FOgug/yk0YzfOjnA7S5sTYWRI49UpLAsZXlhTiUDCofLi7gje2OkngvcKDBBX6pCbyyVOyOc99DeoT480SwaDTWbvLbabZtQIOu1dFp/aFEWfCaV+GkWfDkGlHL1FsaoCKYieBHNwJB3cdnd/0snMpZqjkdCdB9OrvXGHTkgTGWxIwUceSQpyobWhiyYYSlm3Zz5vbyjhQXY8InDqsL7edP5bzxg9kWk5fEgMxXBtorIe964/WAgpXHL2WPWMojDzTSQQjz4LsiZDQzrEIJMGAXOfVlroqN0m4iaJ5vroUTv2CmwTOtiRgopIlhTiwfHsZd7+4gY/LqslMS26pCXxyXBaZ6Sl+h+e9mgPw9i9h9WNHr3HPHAsTP+skgBFnOr/ku6vvIyUdBk12Xsb0MJYUYtjB6np+vHgTC9cWMzKzN4/f8AnOHZcdP1f61B2GFb+F5b9xksGUK2DCJU4iSB/od3TGRCVLCjFIVXnxvd385983UXmkga+fP4ZvXDAupoZ76FBDLax5DN76hdO+P+EzcMEPYaC3Y8YYEwssKcSYnWXV/OCvG3i7oIzpI/rxwOVTmTC4j99hRUZTI6x/Gpb9FCqLnTt5L7gHcmb4HZkxPYYlhRjR0BTk4Td38Oul20gOJPAfn5vCl2aOiI+momAQNi2C137sXEI6bAZ87rcw+jy/IzOmx7GkEAPW7jrIXf/3IVv2HWbOlMHcd+nk2L+cFJz7CrYvhaX3O1cVZU+Aq/7i9Bv05BvmjPGRJYUerLK2gf9aspm/rCpkSJ9UHr02nwsnxclljoWrYOmPnHsM+o2Ay37vDP8Q7SNgGhPlLCn0QKrKkg0l3PvSR5RV1XH9WaP4zqfHk54SB/+cJRvgtf+ArUuccfHnLoDp1/k39r0xMSYOziKxZfehI9y7aAP/2rSfSUP68Oh1+Zya4+3j+aJCfTX87Vvw4fOQ2gdm3wOn32IjgRrTzSwp9BCNTUGeWLGLX/xzC6pw19wJ3Hh2bmzfgRzqg2fhw+fgrNvhk992howwxnQ7Swo9wLIt+3lg8Sa27qti1vhs/mPeFIYP6O13WJFVuNJpLvrU/daJbIyHLClEsc0llfz475t4a1sZIzN787svTefiKYN79lDUJ6pwJYw43RKCMR6zpBCF9h+u5Zf/3Mpza4pIT0nkB5dM5CtnjiQlMU6vrDlc4gwqN3O+35EYE/MsKUSRI/VNPPrWDn73xnbqG4Ncd9Yobr9gHP3T4vzKmsKVznTEGf7GYUwcsKQQBYJBZ6yiBf/cwt6KWi6aPIg750wkN8uurAGgaBUkpjqPojTGeMqSgs9WbC/nx4s3smF3Jafm9OW/r8rj9NGZfocVXQpXOkNX2L0IxnjOkoJPdpRW8ZN/bObVjfsY2jeV/74qj0unDY2PsYq6or4GSj5wLkU1xnjOkkKEHaiu59dLt/HnlbtITQrw3YvGc9M5ufEzrHVX7V4LwUbrTzAmQiwpREh9Y5DHl3/Mb14roLqukWtmjuBbF55CdkYcPPnsZBS5ncw5n/A3DmPihCWFCDhQXc8tT67l3Z0HmDU+m7vmTuSUQRl+h9UzFK5yRj/tPcDvSIyJC5YUPLZt32FufGI1+yrr+NXVeczLG+Z3SD1HMAjF78Kkz/kdiTFxw5KCh17fsp/bn3qP1OQAz33tTPKGx8HAdd2pdDPUVlh/gjERZEnBA6rKY+/s5Md/38hEdyTTIX17+R1Wz9PcnzD8dH/jMCaOWFLoZvWNQe59aQNPv1vExZMH88urptE72Q7zCSlcBWnZMGC035EYEzfsbNWNDlbXc+tf1rJyxwG+fv4YvvOp8XbfwckoWunUEmwQPGMixtPB+EXkYhHZIiIFInJnG9tHishSEflARJaJSI6X8XipYH8Vl/32HdbtOsSDV03juxdNsIRwMg7vg4M7rT/BmAjzLCmISAB4CJgDTAKuEZFJrYotAP6kqqcC9wM/8SoeL721rZTLfvsOVXWNPD3/DC47rcfmtujR0p9gScGYSPKypjATKFDVHapaDzwDzGtVZhKw1J1/vY3tUe9PK3Zy/R9XM6xfL/769bOZMdKeCNYtCt1B8IZM8zsSY+KKl0lhGFAUslzsrgu1Hvi8O38ZkCEix40GJyLzRWSNiKwpLS31JNiuamgK8sO/buCeRR9x/viBLLz1LHL6x9nT0LxUtBKGTrdB8IyJMC+TQlsN6tpq+d+B80TkPeA8YDfQeNybVB9W1XxVzc/Ozu7+SLuooqaBG/64midX7uJr543m91+ZQXqK9dl3m/oa2LveedKaMSaivDyTFQPDQ5ZzgD2hBVR1D3A5gIikA59X1QoPYzppH5dVc9Pjqyk6WMPPrziVK/OHd/4m0zXNg+BZf4IxEedlUlgNjBORXJwawNXAF0MLiEgWcEBVg8D3gcc8jOekLS8o49a/rCOQIDx18xl8YpSNx+OJlk7mmf7GYUwc8qz5SFUbgduAV4BNwHOq+pGI3C8il7rFZgFbRGQrMAj4sVfxnKy3tpVy7WPvMqhPCou+frYlBC8VroKs8TYInjE+8LQhXFUXA4tbrbsnZH4hsNDLGLrLw2/uYGBGCi/cehYZqUl+hxO7WgbB63EXohkTEzy9eS1W7K04wtsFZVwxI8cSgteaB8Gz/gRjfGFJIQz/t243qvD5GXZTmuea+xPsTmZjfGFJoROqysK1xczMHcDIzDS/w4l9haugd5YNgmeMTywpdGLtroN8XFbNFVZLiIyilU4twQbBM8YXlhQ68fyaYnonB7hk6hC/Q4l9zYPg2fMTjPGNJYUO1NQ38vcP9zJ36hDS7I5l71l/gjG+s6TQgSUbSqiqa7Smo0gpXAWBFBsEzxgfWVLowPNrihkxoDen59pNVBFRtBKGTYfEFL8jMSZuWVJoR9GBGlbsKOeKGTmIdXp6r3kQPOtPMMZXlhTa8cK6YkTg8umtR/s2ntizzhkEz/oTjPGVJYU2BIPKC+uKOWtMpj0jIVIKmwfBs5qCMX6ypNCGVR8foOjAEa6cYcNiR0zRKsg6xQbBM8ZnlhTa8PzaIjJSErlo8mC/Q4kPwaCTFKyWYIzvLCm0UlXXyD8+LOEz04bQKzngdzjxoWyLMwie9ScY4ztLCq0s/mAvRxqauMKajiKnpT/BkoIxfrOk0MrCtcWMzk5j+oh+focSP4rcQfAyx/gdiTFxz5JCiJ1l1by784DdmxBphSud/gQ75sb4zpJCiIVri0kQuPw0G9YiYqr2w8GPYYR1MhsTDSwpuJrcexM+OS6bwX1T/Q4nflh/gjFRxZKCa/n2MvZW1HJlvtUSIqrIHQRvaJ7fkRhjsKTQYuHaYvqkJnLhxEF+hxJfClfC0NNsEDxjooQlBaDiSANLNpQwL28YqUl2b0LENBxxBsGz/gRjooYlBeDlD/ZQ1xi0pqNI270Ogg3Wn2BMFLGkgNN0dMqgdKYO6+t3KPGlyAbBMybaxH1SKNh/mPcKD3HljOF2b0KkFa6CzHGQlul3JMYYV9wnhefXFhNIED53mj03IaKaB8Gz/gRjokpcJ4XGpiAvrtvN+eOzyc6wq18iqmwr1B6y/gRjokxcJ4W3tpWx/3CdDX7nh+b+BBsZ1ZioEtdJYeHaYgakJXPBhIF+hxJ/CldB70zIHOt3JMaYEHGbFA7V1PPqxn3MyxtKcmLcHgb/FK10mo6sc9+YqOLp2VBELhaRLSJSICJ3trF9hIi8LiLvicgHIjLXy3hCLXp/D/VNQXvkph+q9sOBHdbJbEwU8iwpiEgAeAiYA0wCrhGRSa2K/QB4TlVPA64GfutVPK0tXFvMpCF9mDS0T6Q+0jQrWuVMrZPZmKjjZU1hJlCgqjtUtR54BpjXqowCzWflvsAeD+Npsbmkkg93V9gdzH4pXGmD4BkTpbxMCsOAopDlYnddqPuAL4tIMbAY+EZbOxKR+SKyRkTWlJaWnnRgC9cUkxQQ5uXZvQm+KFplg+AZE6W8TApt9SBqq+VrgMdVNQeYCzwpIsfFpKoPq2q+quZnZ2efVFANTUH++v5uZk8YxIC05JPalzkBDUdgz/vWn2BMlPIyKRQDob24ORzfPHQT8ByAqq4AUoEsD2Pi9c37Kauqt6Yjv+x5zwbBMyaKeZkUVgPjRCRXRJJxOpJfalWmEJgNICITcZLCybcPdWDh2mKy0lM475STq3GYE1Rog+AZE808Swqq2gjcBrwCbMK5yugjEblfRC51i30HuFlE1gNPA9erausmpm5TVlXHa5v3c/n0YSQG7N4EXxTZIHjGRLNEL3euqotxOpBD190TMr8RONvLGEL99b3dNAaVK2dY05EvmgfBm3CJ35EYY9oRNz+XVZWFa4uZNrwf4wZl+B1OfCrfBkcOWn+CMVGs06QgIreJSP9IBOOlj/ZUsrnkMFdYLcE/hTYInjHRLpyawmBgtYg85w5b0SMHq1m6aT/JiQlceupQv0OJX0U2CJ4x0a7TpKCqPwDGAX8Arge2icgDIjLG49i61e2zx/LqHefSt3eS36HEr8KVzlVHPfN3hTFxIaw+BfeKoBL31Qj0BxaKyH95GFu3EhFGZqb5HUb8qiqFA9vtUlRjolynVx+JyO3AdUAZ8CjwXVVtcO883gb8P29DNDHBHqpjTI8QziWpWcDlqrordKWqBkXkM96EZWLO9tcgOR2GTvc7EmNMB8JpPloMHGheEJEMETkdQFU3eRWYiSGqULAURn0SEm28KWOiWThJ4XdAVchytbvOmPAc2AGHdsHY2X5HYozpRDhJQUKHnlDVIB7fCW1iTMFSZzrmAn/jMMZ0KpyksENEbheRJPf1TWCH14GZGLL9Neg/CjJ71FXMxsSlcJLCLcBZwG6c4bBPB+Z7GZSJIY31sPMtqyUY00N02gykqvtxhr02puuKVkF9FYyx/gRjeoJw7lNIxXkYzmSc5x0AoKo3ehiXiRXbl0JCIuSe63ckxpgwhNN89CTO+EcXAW/gPEHtsJdBmRhSsBRyZkJqH78jMcaEIZykMFZVfwhUq+oTwCXAVG/DMjGhaj+UfABjrT/BmJ4inKTQ4E4PicgUoC8wyrOITOzY/roztf4EY3qMcO43eNh9nsIPcJ6xnA780NOoTGzYvtQZKntInt+RGGPC1GFScAe9q1TVg8CbwOiIRGV6vmDQuT9h9PmQEDcP+DOmx+vwf6t79/JtEYrFxJJ9H0J1qQ1tYUwPE85PuFdF5N9FZLiIDGh+eR6Z6dlsaAtjeqRw+hSa70f4esg6xZqSTEe2vwaDpkDGYL8jMcZ0QTh3NOdGIhATQ+qqnEdvnnGL35EYY7oonDuar21rvar+qfvDMTFh59sQbLBLUY3pgcJpPvpEyHwqMBtYB1hSMG3bvhQSe8GIM/2OxBjTReE0H30jdFlE+uIMfWFM2wqWwqhzICm187LGmKhyIheQ1wDjujsQEyMO7oQD2+1SVGN6qHD6FP6Gc7UROElkEvCcl0GZHqzlUlRLCsb0ROH0KSwImW8EdqlqsUfxmJ5u+2vQdzhkWWXSmJ4onKRQCOxV1VoAEeklIqNUdaenkZmep6kBdrwBUy4HEb+jMcacgHD6FJ4HgiHLTe66TonIxSKyRUQKROTONrY/KCLvu6+tInIovLBNVCpeDfWHrT/BmB4snJpCoqrWNy+oar2IJHf2JhEJAA8Bn8J5tvNqEXlJVTeG7OuOkPLfAE7rSvAmyhQsBQlA7nl+R2KMOUHh1BRKReTS5gURmQeUhfG+mUCBqu5wk8ozwLwOyl8DPB3Gfk202r4UcvKhVz+/IzHGnKBwksItwF0iUigihcD3gK+F8b5hQFHIcrG77jgiMhLIBV5rZ/t8EVkjImtKS0vD+GgTcdXlsOd9GwDPmB4unJvXtgNniEg6IKoa7vOZ2+pp1DbWAVwNLFTVpnZieBh4GCA/P7+9fRg/7XgdULsU1ZgertOagog8ICL9VLVKVQ+LSH8R+c8w9l0MDA9ZzgH2tFP2aqzpqGfb/hqk9oNh0/2OxBhzEsJpPpqjqi1XBblPYZsbxvtWA+NEJNftmL4a53GexxCR8UB/YEV4IZuoo+o+ZW0WJAT8jsYYcxLCSQoBEUlpXhCRXkBKB+UBUNVGnKe2vQJsAp5T1Y9E5P7QjmucDuZnVNWahXqq/Rvh8F67FNWYGBDOJal/BpaKyB/d5RuAJ8LZuaouBha3WndPq+X7wtmXiWI2tIUxMSOcjub/EpEPgAtxOo+XACO9Dsz0INuXQvYE6NvmxWXGmB4k3FFSS3Duav48zvMUNnkWkelZ6mtg1wqrJRgTI9qtKYjIKTidw9cA5cCzOJeknh+h2ExPsOsdaKqDsXZ/gjGxoKPmo83AW8BnVbUAQETu6KC8iUcFSyExFUae7Xckxphu0FHz0edxmo1eF5FHRGQ2bd+QZuLZ9qUw8ixI6uV3JMaYbtBuUlDVF1X1KmACsAy4AxgkIr8TkU9HKD4TzQ4VQdlWG9rCmBjSaUezqlar6l9U9TM4dyW/Dxw3DLaJQ9vtUlRjYk2XntGsqgdU9feqaj8NjXMXc8ZQGDjR70iMMd2kS0nBmBZNjbBjmdN0ZE9ZMyZmWFIwJ2bPOqitsEtRjYkxlhTMiSlYCgiMtttWjIkllhTMidm+1Bkmu/cAvyPfBSlsAAAR8klEQVQxxnQjSwqm644chN1r7aojY2KQJQXTdTuWgQZtqGxjYpAlBdN1BUshpS8My/c7EmNMN7OkYLqm5Slr50IgnMdxGGN6EksKpmtKt0DlbhvawpgYZUnBdI0NbWFMTLOkYLqmYClkjoX+9vA9Y2KRJQUTvoZa56E6VkswJmZZUjDhK1wOjbV2KaoxMcySgglfwVIIJMOoc/yOxBjjEUsKJnzbX4MRZ0Bymt+RGGM8YknBhKdyD+zfaP0JxsQ4SwomPBtecKZjL/Q3DmOMpywpmM7VVsBbv4TRs2DwFL+jMcZ4yJKC6dw7v4YjB+DCH/kdiTHGY5YUTMcq98KKh2DKFTA0z+9ojDEes6RgOvbGTyHYCBf8wO9IjDER4GlSEJGLRWSLiBSIyJ3tlPmCiGwUkY9E5Ckv4zFdVLoV1j0J+TfCgFy/ozHGRIBnYx+LSAB4CPgUUAysFpGXVHVjSJlxwPeBs1X1oIgM9CoecwKW/giSesN5/8/vSIwxEeJlTWEmUKCqO1S1HngGmNeqzM3AQ6p6EEBV93sYj+mKondh88tw9u2QluV3NMaYCPEyKQwDikKWi911oU4BThGRd0RkpYhc7GE8Jlyq8Oq9kDYQzvg3v6MxxkSQl4/OkjbWaRufPw6YBeQAb4nIFFU9dMyOROYD8wFGjBjR/ZGaY219xRn87pJfQkq639EYYyLIy5pCMTA8ZDkH2NNGmUWq2qCqHwNbcJLEMVT1YVXNV9X87OxszwI2QLAJ/nUfDBgD06/1OxpjTIR5mRRWA+NEJFdEkoGrgZdalfkrcD6AiGThNCft8DAm05n1T0PpJph9DwSS/I7GGBNhniUFVW0EbgNeATYBz6nqRyJyv4hc6hZ7BSgXkY3A68B3VbXcq5hMJxqOwOsPwLAZMKn1NQHGmHjgZZ8CqroYWNxq3T0h8wp8230Zv737MFTuhssfBmmrS8gYE+vsjmbjqDkAb/0Cxn3aHqJjTByzpGAcbz8ItZUw+16/IzHG+MiSgoGKYlj1e5h2tQ2NbUycs6Rg4PWfONPz7/Y3DmOM7ywpxLt9G2H9UzDzZug3vPPyxpiYZkkh3i39ESRnwCe/43ckxpgoYEkhnu18B7YugXO+Bb0H+B2NMSYKWFKIV6rwr3shYyiccavf0RhjooQlhXi16W9QvBrO/z4k9fI7GmNMlLCkEI+aGp2+hKzxMO2LfkdjjIking5zYaLUe09CeQFc/TQE7CtgjDnKagrxpr4alv0Uhp8B4+f4HY0xJsrYz8R4s/K3UFUCX3jCBr0zxhzHagrxpLoc3v4VjL8ERpzhdzTGmChkSSGevLUAGqrhQhv0zhjTNms+ihelW+DdR+C0L0P2eL+jMSYiGhoaKC4upra21u9QIiY1NZWcnBySkk7syYmWFOLBxkWw6DZISYdZ3/c7GmMipri4mIyMDEaNGoXEQR+aqlJeXk5xcTG5ubkntA9rPopljfXwj+/Bc9dC1jj42pvQZ6jfURkTMbW1tWRmZsZFQgAQETIzM0+qZmQ1hVh1cBc8fz3sWQdn/Btc+CNITPY7KmMiLl4SQrOT/XstKcSizX+Hv94KClz1Z5j4Wb8jMsb0ENZ8FEuaGuCVu+GZL0L/XPjaG5YQjPFRenr6ceu2bNnCrFmzyMvLY+LEicyfP59XXnmFvLw88vLySE9PZ/z48eTl5XHttdeybNkyRIQ//OEPLft47733EBEWLFjQ7TFbTSFWVBTD8zdA8bvwiZvhoh9DYorfURljWrn99tu54447mDdvHgAffvghU6dO5aKLLgJg1qxZLFiwgPz8fACWLVvG1KlTefbZZ7npppsAeOaZZ5g2bZon8VlSiAVb/wkvzncGurvijzDlcr8jMibq/OhvH7FxT2W37nPS0D7c+9nJXXrP3r17ycnJaVmeOnVqp+8ZMWIElZWV7Nu3j4EDB7JkyRLmzp3b5XjDYc1HPVlTI7x6Lzx1JfTNcZqLLCEYE9XuuOMOLrjgAubMmcODDz7IoUOHwnrfFVdcwfPPP8/y5cuZPn06KSnetARYTaGnqtwDC2+EwhUw4wa4+KeQlOp3VMZEra7+ovfKDTfcwEUXXcSSJUtYtGgRv//971m/fn2nJ/kvfOELXHXVVWzevJlrrrmG5cuXexKf1RR6ooKl8L/nwN4P4PJH4bP/bQnBmB5k6NCh3HjjjSxatIjExEQ2bNjQ6XsGDx5MUlISr776KrNnz/YsNqsp9CTBJlj2E3hzAQycCFc+Admn+B2VMaYLlixZwuzZs0lKSqKkpITy8nKGDRsW1nvvv/9+9u/fTyAQ8Cw+SwqREgw6Q1Y31jmXjgYboKne6RdoqneXm1/1EHTXhy5v/jvsfMsZv2jOzyG5t99/lTGmAzU1Ncd0Kn/729+muLiYb37zm6SmOrX7n//85wwePDis/Z111lmexBlKVNXzD+lO+fn5umbNGr/D6FxTg9O8s+sdp92/cAUcOXhy+0xKg0sWQJ49QtOYcGzatImJEyf6HUbEtfV3i8haVc3v7L1WU+gu9TVQvNo5+e9a7sw31DjbBox2nmEwNA+S0yCQDAmJzjSQ7DwSM5AMCUkQSHLXufMJSUfLJKXZUBXGGE95mhRE5GLgV0AAeFRVf9pq+/XAz4Hd7qr/UdVHPQmmbJszfHSvfpDaD1L7OvPJ6Sf2BLKaA1C0ykkAhStgz3tOEw8Cg6fAaV+BkWfCiDMhI7yqoTHG+M2zpCAiAeAh4FNAMbBaRF5S1Y2tij6rqrd5FUeLzS/Dv+47fr0EjiaI1H7utG/b802NULTSSQT73T8jkAxDp8NZ34ARZ8HwmU5ZY4zpgbysKcwEClR1B4CIPAPMA1onhciYfh2MPh9qK6D2EBw51GreXT5yCA4VHp0PNhy7n+R058Q/+XKnJjBsBiT18uVPMsaY7uZlUhgGFIUsFwOnt1Hu8yJyLrAVuENVi1oXEJH5wHxwbvc+Ib0HOK+uUHX6BZoThAYhe4LTvm+MMTHIy5vX2mqob32p09+AUap6KvAv4Im2dqSqD6tqvqrmZ2dnd3OYHRBxOob7DIVBk5y+AksIxpgY5mVSKAaGhyznAHtCC6hquarWuYuPADM8jMcYYyKupKSEq6++mjFjxjBp0iTmzp3L1q1bERF+85vftJS77bbbePzxxwG4/vrrGTZsGHV1zumxrKyMUaNGRSReL5PCamCciOSKSDJwNfBSaAERGRKyeCmwycN4jDEmolSVyy67jFmzZrF9+3Y2btzIAw880DLa6a9+9Svq6+vbfG8gEOCxxx6LcMQe9imoaqOI3Aa8gnNJ6mOq+pGI3A+sUdWXgNtF5FKgETgAXO9VPMaYOPePO6Hkw+7d5+CpMOen7W5+/fXXSUpK4pZbbmlZl5eXx86dO8nOzubss8/miSee4Oabbz7uvd/61rd48MEH29zmJU8HxFPVxap6iqqOUdUfu+vucRMCqvp9VZ2sqtNU9XxV3exlPMYYE0kbNmxgxoz2W8XvvPNOfvGLX9DU1HTcthEjRnDOOefw5JNPehnicazX1BgTHzr4Re+X3NxcZs6cyVNPPdXm9rvuuotLL72USy65JGIx2dDZxhjjkcmTJ7N27doOy9x111387Gc/IxgMHrdt7Nix5OXl8dxzz3kV4nEsKRhjjEcuuOAC6urqeOSRR1rWrV69ml27drUsT5gwgUmTJvHyyy+3uY+7776bBQsWeB5rM0sKxhjjERHhxRdf5NVXX2XMmDFMnjyZ++67j6FDhx5T7u6776a4uLjNfUyePJnp06dHIlzAhs42xsQwGzr7qHCHzraagjHGmBaWFIwxxrSwpGCMiWk9rYn8ZJ3s32tJwRgTs1JTUykvL4+bxKCqlJeXtzz/+UTYzWvGmJiVk5NDcXExpaWlfocSMampqeTk5Jzw+y0pGGNiVlJSErm5uX6H0aNY85ExxpgWlhSMMca0sKRgjDGmRY+7o1lESoFdnRb0RxZQ5ncQHbD4Tk60xwfRH6PFd3JOJr6Rqtrp84x7XFKIZiKyJpzbyP1i8Z2caI8Poj9Gi+/kRCI+az4yxhjTwpKCMcaYFpYUutfDfgfQCYvv5ER7fBD9MVp8J8fz+KxPwRhjTAurKRhjjGlhScEYY0wLSwpdJCLDReR1EdkkIh+JyDfbKDNLRCpE5H33dU+EY9wpIh+6n33cY+rE8WsRKRCRD0QkYs/6E5HxIcflfRGpFJFvtSoT8eMnIo+JyH4R2RCyboCIvCoi29xp/3bee51bZpuIXBeh2H4uIpvdf78XRaRfO+/t8LvgcYz3icjukH/Hue2892IR2eJ+H++MYHzPhsS2U0Teb+e9nh7D9s4pvn3/VNVeXXgBQ4Dp7nwGsBWY1KrMLOBlH2PcCWR1sH0u8A9AgDOAVT7FGQBKcG6q8fX4AecC04ENIev+C7jTnb8T+Fkb7xsA7HCn/d35/hGI7dNAojv/s7ZiC+e74HGM9wH/HsZ3YDswGkgG1rf+/+RVfK22/wK4x49j2N45xa/vn9UUukhV96rqOnf+MLAJGOZvVF02D/iTOlYC/URkiA9xzAa2q6rvd6ir6pvAgVar5wFPuPNPAJ9r460XAa+q6gFVPQi8ClzsdWyq+k9VbXQXVwInPlZyN2jn+IVjJlCgqjtUtR54Bue4d6uO4hMRAb4APN3dnxuODs4pvnz/LCmcBBEZBZwGrGpj85kisl5E/iEikyMaGCjwTxFZKyLz29g+DCgKWS7Gn8R2Ne3/R/Tz+DUbpKp7wfmPCwxso0w0HMsbcWp+bensu+C129wmrsfaaf6IhuP3SWCfqm5rZ3vEjmGrc4ov3z9LCidIRNKBF4BvqWplq83rcJpEpgG/Af4a4fDOVtXpwBzg6yJybqvt0sZ7InptsogkA5cCz7ex2e/j1xW+HksRuRtoBP7STpHOvgte+h0wBsgD9uI00bTm+3cRuIaOawkROYadnFPafVsb607q+FlSOAEikoTzj/cXVf2/1ttVtVJVq9z5xUCSiGRFKj5V3eNO9wMv4lTRQxUDw0OWc4A9kYmuxRxgnarua73B7+MXYl9zs5o73d9GGd+Opdup+BngS+o2MLcWxnfBM6q6T1WbVDUIPNLOZ/v6XRSRROBy4Nn2ykTiGLZzTvHl+2dJoYvc9sc/AJtU9ZftlBnslkNEZuIc5/IIxZcmIhnN8zgdkhtaFXsJuNa9CukMoKK5mhpB7f468/P4tfIS0Hw1x3XAojbKvAJ8WkT6u80jn3bXeUpELga+B1yqqjXtlAnnu+BljKH9VJe189mrgXEikuvWHq/GOe6RciGwWVWL29oYiWPYwTnFn++fVz3qsfoCzsGpnn0AvO++5gK3ALe4ZW4DPsK5kmIlcFYE4xvtfu56N4a73fWh8QnwEM5VHx8C+RE+hr1xTvJ9Q9b5evxwEtReoAHn19dNQCawFNjmTge4ZfOBR0PeeyNQ4L5uiFBsBThtyc3fwf91yw4FFnf0XYjg8XvS/X59gHOCG9I6Rnd5Ls4VN9u9irGt+Nz1jzd/70LKRvQYdnBO8eX7Z8NcGGOMaWHNR8YYY1pYUjDGGNPCkoIxxpgWlhSMMca0sKRgjDGmhSUFY1oRkSY5diTXbhu5U0RGhY7UaUy0SfQ7AGOi0BFVzfM7CGP8YDUFY8Lkjqv/MxF5132NddePFJGl7sBvS0VkhLt+kDjPOljvvs5ydxUQkUfcsfP/KSK9fPujjGnFkoIxx+vVqvnoqpBtlao6E/gf4L/ddf+DMxT5qTgD0/3aXf9r4A11BvabjnNHLMA44CFVnQwcAj7v8d9jTNjsjmZjWhGRKlVNb2P9TuACVd3hDmBWoqqZIlKGM4RDg7t+r6pmiUgpkKOqdSH7GIUz/v04d/l7QJKq/qf3f5kxnbOagjFdo+3Mt1emLXUh801Y356JIpYUjOmaq0KmK9z55TijewJ8CXjbnV8K3AogIgER6ROpII05UfYLxZjj9ZJjH+K+RFWbL0tNEZFVOD+ornHX3Q48JiLfBUqBG9z13wQeFpGbcGoEt+KM1GlM1LI+BWPC5PYp5Ktqmd+xGOMVaz4yxhjTwmoKxhhjWlhNwRhjTAtLCsYYY1pYUjDGGNPCkoIxxpgWlhSMMca0+P/dTHyBfqZMGAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x268ec24e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Training Accuracy\n",
    "x = np.arange(20)+1\n",
    "\n",
    "plt.plot(x, model_hist_embedding.history['acc'])\n",
    "plt.plot(x, model_hist_cnn.history['acc'])\n",
    "plt.legend(['LSTM', 'CNN'], loc='lower right')\n",
    "plt.ylabel(\"Accuracy\")\n",
    "axes = plt.gca()\n",
    "axes.set_ylim([0.45,1.01])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.title(\"Training Accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xl8HWXZ//HPla1pm+57m+6UrkAplX2TvSxFBIWKIqAiKoK4Ijwq4vITxMcVVBAelocdFQtPAQFZRARboC1dKF3oEtq0abo3TbNdvz/uyelpOElO2sw5Sft9v17ndebM3DNzncnJXHPfM3OPuTsiIiIAOdkOQERE2g4lBRERSVBSEBGRBCUFERFJUFIQEZEEJQUREUlQUpD9gpl1MLNtZjYw27GItGVKCpJV0Y66/lVnZjuSPl+8F8t93cw+Xf/Z3Xe6e5G7r26dyFOu80ozczObGtc6ROKmpCBZFe2oi9y9CFgJnJM07oFsx9dCnwU2RO8ZZWa5mV6n7JuUFKRNM7NcM/uemS0zs/Vm9oCZdY+mdTazh81sg5ltMrM3zKyHmf0C+Ajwp6jG8QszK4yO4oujeR82s1+Z2bNmttXM/mVmQ5PWe5aZLY6W+6uGNY8UcR4IHAF8ETjbzHo2mP4JM5trZlui5Z4cje9tZveZWamZbTSzR6LxV5rZ80nzp4r/N2b2dzPbDhxlZueZ2ZxoHSvM7PoGMZwYfY/NZrbSzD5lZseZ2Sozy0kqd7GZvb5nfzFp75QUpK37FnAacCxQDFQDv4ymfR7IAwYBvYGrgCp3/wYwE/h8VOP4RiPL/hTwXaAnsAb4IYCZ9QceAa4F+gCrgcOaifOzwKvu/jiwAriofoKZHQfcAVwDdAdOBlZFkx8BDBgD9ANua2Y9yT4NfA/oQvi+W6Lv1B04D/immZ0RxXAA8BTwc6BX9H3mu/s/gSrghAbLvb8Fccg+RElB2rovAte5+2p3ryTsuC80MyMkiD7ASHevcfeZ7r69Bct+1N3fcvdq4EFgYjR+KjDT3Z+Kpt0KbGxsIdFR9meiZQA8xO5NSJ8H/uDuL7p7nbuvdPf3zGw4cBzwZXff5O5V7v5KC+J/3N3fiJa5091fcPf50ee3gEfZtbP/DPCku/852lZl7j4nmnYfIRFgZv2ieR5pQRyyD1FSkDYr2vEPBmZEzTibgLcJv9tewF3Ay8DjZlZiZj9tYdt6adJwBVAUDQ9k15E87l4HfNDEcj4K9Aceiz4/ABxuZmOiz4OBpSnmGwysc/etLYg52arkD2Z2jJm9bGZlZrYZuJRQg2oqBghJ4eNmVghMA55z9/V7GJO0c0oK0mZ56ML3A+Akd++e9Cp09/XR0fH33X0McDzwCXY12+xN979rCE1VQKImMKiJ8p8l/C/NN7NS4JVo/ZdE01cBI1PMtwroa2ZFKaZtBzolfe6fokzD7/go4Qh/sLt3A+4hNE01FQPu/j4wFziHUKNQ09F+TElB2ro/AD8zs8EAZtbXzM6Jhk8xs3HRTnsLUAPURvOtBUbs4TqnA0eY2Zlmlgd8HeiRqmC0Q/844ah8YtLrm8Bnotj+BHzRzI43sxwzG2xmB0Y741eA35lZNzMrMLPjo0XPBg41s/Fm1gn4flMBR7WqIqDc3SvN7GhCkqx3H+EE+HnRyfs+ZnZwg+nfI2yzJ9PcTrIPUlKQtu4W4HngH2a2FXgNmBRNGwT8DdgKzANmEI6WIZyMviS6oueWlqzQ3dcQmlF+A6wn1BreAXamKH4B4TLUh9y9tP5FOLHclVDL+SdwJXA7sBl4gV01kWlAPrCY0Jz1pSiGd6Lv/k/gXeClZmL2aB23Rtvp2+xqzsLdlwLnAtcTzo/MAsYnLeIx4ADCeZZU31P2E6aH7Ig0LaotlBLuofh3tuOJQ1SjWQlc5O6vZjseyR7VFERSMLMpUZNOIfADwonoN7McVpymAVuUECQv2wGItFHHE64iyiM0TZ3n7lXZDSke0Y1qwwj3OMh+Ts1HIiKSoOYjERFJaHfNR7179/Zhw4ZlOwwRkXblzTffXO/ufZorF1tSMLO7gbMJd2xOSDHdgF8DZxJO4l0a3ZrfpGHDhjFr1qzWDldEZJ9mZivSKRdn89E9wBlNTJ8CjIpeVwC/jzEWERFJQ2xJIerYa0MTRc4F7vPgdaC7mQ2IKx4REWleNk80D2L3Dr1KaKR/GTO7wsxmmdmssrKyjAQnIrI/ymZSsBTjUl4f6+53uPtkd5/cp0+z50lERGQPZTMplBC6861XTHiYiYiIZEk2k8J0QodlZmZHApujjshERCRL4rwk9SHgRKC3mZUQ+o/JB3D3PxB6tDwTWEK4JPWyuGIREZH0xJYU3H1aM9Md+Epc6xcRkZZTNxciIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJsSYFMzvDzBaZ2RIzuy7F9KFm9oKZzTWzl8ysOM54RESkabElBTPLBW4DpgDjgGlmNq5BsVuB+9z9YOAm4P/FFY+IiDQvzprC4cASd1/m7lXAw8C5DcqMA16Ihl9MMV1ERDIozqQwCFiV9LkkGpdsDnB+NHwe0MXMejVckJldYWazzGxWWVlZLMGKiEi8ScFSjPMGn78JnGBmbwMnAB8ANR+ayf0Od5/s7pP79OnT+pGKiAgAeTEuuwQYnPS5GFidXMDdVwMfBzCzIuB8d98cY0wiItKEOGsKM4FRZjbczAqAi4DpyQXMrLeZ1cfwXeDuGOMREZFmxJYU3L0GuAp4FlgIPOru883sJjObGhU7EVhkZu8B/YCfxBWPiIg0z9wbNvO3bZMnT/ZZs2ZlOwwRkXbFzN5098nNldMdzSIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglxPk+hTdlcUU1VbR29iwowS/X8n7bt/fXbeWnROjoX5NG1Yx5dO+bTtTCfbh3z6dYpn6KCPHJy2t/3EpG2Zb9JCo/OWsVPZiykY34uxT06MqRnJwb37LTb8OCenSjq0LY2yYLVW7j9pSXMeGcNdU10aJtj0CVKEl075oX3+qTRMZ+uHfPp0amAYb07MbJPEX27dGhXyfG1Jev5n9eWM6xXJw4d0oOJg7szoFthu/oOsncqqmqoc8jPNfJzclr1IKiqpo4dVbVUVNdQUVUbhqtq2V5VkxjeUV1LVU0dNbV1VNfWUV3r0fuu4Zr6cXVOdU0dNXV1VNV6Yp699fnjRnD6+P6t8I0b17b2gDE67sDe3Jg7jlUbd7BqQwUrN1Twxvsb2LZz96d/9uxcwOAeHSnu2Skkix6dGNwzJI6B3TuSn5uZFrc3V2zktheX8I9311HUIY8vnjCSi48YAsCWHTVs3lHN5h3VbKmsZsuO8No1Lkxfu2VbYvzOmt1/kEUd8hjeuzPDe3dmRJ/OjOhTxIhouFNB2/lZuDt/fGUZtzzzLj06FfDye2Xc+c/3AejXtQMTB3dPJImDi7u1qdhlz1RW17J47TYWrd3Ke2u38m7pVt4r3UrplsrdyuXmWCJB5OflkJdj5OfmhHG5ObsN5+UaeTk57KypTbnTr2nqiKsJ9evMyzUKonXWD+cl1p1DQbT+vT2Gyc3AQdB+/TwFd2dTRTUrN1SwamMFqzbsYOWGCko2VrBqQwUfbNpBde2u7VOYn8OJB/ZlykH9OWlMX7oU5rdKHMnxvLpkPbe9uITXl22gR6d8PnfscD5z1DC6ddy7dVVW11K+vYr3y7azbP02lpVtZ2lZeF+9eQfJP4MB3QpDouhdxIg+IXGM7FPEwO4dyc1gE9XWymq+9dhcnplfylkHDeDmCw6mIDeHhWu2MHvVJt5euZHZqzaxvLwCCDuJA/t14dAh3Zk4uDuThnRnRO+iPTqirK3zkGwrdyXbyuo6hvfuxLBencnL0MHBvqymto7l5RW77fgXrd3KivLtiVpxQV4Oo/oWMbpfF0b2LSIvx6ip83DEXrf70XpNrVOVfLRef/ReV0d1jVNTV0eHvFw6FeTSsSC8dyrIo2NBLp0LculYkBeNy6Vj/q5piXEFuXTIzSU/L+zg83OtXdVU032ewn6dFJpTW+eUbqlk1YaQJOaWbObZ+aWs27qTgtwcjhvVmzMm9OfUcf3o3qlgj9dTV+f8fcFabn9pCXNLNtO/ayFfOH4E0w4fnJEj38rqWpaXb2dZ2XaWRYli6fowvLVyV02qa2EeXzvlQC45amjsO8X31m7lyvvfZMWGCr47ZQyfO3Z4o/+AG7ZXMSdKEm+v2sTsVZsScXcpzAu1icHdGTOgK1U1dbtqVLvVrqrZvKMmUeva2qAGmawgN4eRfYsY3a+IA/t3YUz/LhzYrwuDundsVzuJOLiHHXP9kfiuo/IatlTWsLRsG4tKt7KodCtLyrZRFdVgcwyG9erM6Ghbju4fXkN7dlICbiVKCjGpq3PeXrWRGe+U8sy8Uj7YtIO8HOOokb2YMmEAp43vR++iDmktq6a2jifnrub2F5eyeN02hvbqxJdOGMl5kwbRIS835m/SPHdn/bYqlpVt4/3125kxr5RX3itj7ICu/Phj4zlsaM9Y1vvknNV8589z6VSQy+8+NYkjR/Rq0fx1dc6y9dt4e+WmkCRWbuLd0i0fOifTqSB3t3MvXTvufk6mW4OT+fm5Oby/ftuuo9rSrazevKtJo6hDHgf2K2J0/66Mrn/v34Wenff8gKEtqKmt47Wl5fzj3XVs3lFNRdXu7e47qsNOv2JnLRXVtdQ20xQzoFshB/bblUxH9+/CAX2LKMzP/m9+X6akkAHuztySzTw9r5Rn5q1heXkFOQaHD+/JlAkDOGNCf/p1LfzQfJXVtTz+Zgl/fGUpqzbsYHS/Lnz5oyM566ABbfqoyN15dn4pP3xyAWs2V3Lh5MF8Z8qYVtvpVdfW8bOn3+WuV9/nsKE9uO1Tk+jf7cPbb09UVNWwrGx7IhF0KcynIG/vt/WWyupEs0f9EfCitVvZVFGdKNOnSwdG9+vCiD6dyTGLmjhSn6isTtH8UT9sBkcM78Wp4/py3Kg+dI7xooi6OufNlRuZPns1M95ZQ/n2Kjrm59KrqCBqSsmjc6JZJY9O+bkNmlryoiaZ0AzTqSCXzh3yGN6rM906tW6zq6RHSSHD3J2Fa7byzLw1PD2vlMXrtgFw2NAeTJnQnzMm9KdHpwIefGMld/5zGeu27mTi4O5c9dEDOGlM33Z1Oen2nTX85oXF3PXq+xQV5vHt08dw0UcG79V3WLe1kqseeJv/LN/ApUcP4/ozx7bKTjsb3J2yrTs/lCjeX7+dHLPdTn7m5+ZEJ0p3Hw5t1rufKK2sruVfS8rZvKOagtwcjhrZi1PG9eOUsX0Z0K1jq8Q9f/UWnpyzmqfmruGDTTvokJfDKWP7cc4hAzlxdB8dzbdjSgpZtmTdVp5+p5Sn55WyYM0WADrm57KjupZjDujFV048gKNG9mrXbdDvrd3K956Yxxvvb2Di4O78+GMTmDCoW4uXM2v5Br78wFtsqazmZx8/mI8dOiiGaPcNNbV1zFqxkecXrOW5hWtZEZ1kHz+wK6eM7cep4/oxfmDXFv2ulpVtY/qc1Uyfs5plZdvJyzGOP7APUw8ZyCnj+rW5y7RlzygptCEryrfz9LxSVpRv55OTB3PokB7ZDqnVuDtPzP6An/zfQjZsr+IzRw7l66eNTutqKXfnnteW85P/W0hxj478/tOHMXZA1wxEvW9wd5aWbeP5het4fsFa3ly5EXfo37WQk8f25ZRx/ThqRK+UR/erN+3gySgRzF+9JWqa6snUQwYxZUJ/erTz8yDyYUoKklGbd1Tz339fxP2vr6Bn5w7ccNYYPjZxUKNHrBVVNVz353eYPmc1p4ztxy8+echeX3a7vyvftpN/vLuOFxau45XFZVRU1dKpIJfjRvXmlLH9OGxoD/61ZD3T56xm5vKNABwyuDvnHDyAsw8e2Grnb6RtUlKQrJj3wWZueGIec1Zt4ojhPfnxxyYwql+X3cq8v347V97/JovXbeUbp43mSyeMbFfnVNqDyupa/r2snOcXrOWFhet2u/HrwH5FTD1kIOccMpChvTpnMUrJJCUFyZq6Oufhmau4+Zl32b6zhs8dN5yrTxpF5w55/H1+Kd94dA55ucZvph3KcaP6ZDvcfV79CeS3V27kI8N7Mqa/muj2R0oKknXl23Zy8zPv8uisEgZ2K+S4UX14ZNYqDi7uxu0XT6K4R6dshyiy30g3KbTPa/6kXehV1IFbLjiEx688iq4d83lk1iqmHT6ER794lBKCSBula80kdpOH9eSprx7LsvXbObDB+QURaVtUU5CMyMvNUUIQaQeUFEREJCHWpGBmZ5jZIjNbYmbXpZg+xMxeNLO3zWyumZ0ZZzwiItK02JKCmeUCtwFTgHHANDMb16DYfwGPuvuhwEXA7XHFIyIizYuzpnA4sMTdl7l7FfAwcG6DMg7UXzTdDVgdYzwiItKMOJPCIGBV0ueSaFyyG4FPm1kJMAP4aqoFmdkVZjbLzGaVlZXFEauIiBBvUkjVb0HDO+WmAfe4ezFwJnC/mX0oJne/w90nu/vkPn10B6yISFziTAolwOCkz8V8uHnoc8CjAO7+b6AQ6B1jTCIi0oQ4k8JMYJSZDTezAsKJ5OkNyqwETgYws7GEpKD2IRGRLIktKbh7DXAV8CywkHCV0Xwzu8nMpkbFvgF8wczmAA8Bl3p764xJRGQfEms3F+4+g3ACOXnc95OGFwDHxBmDiIikT3c0i4hIQrNJwcyuMrN95/mRIiLSqHRqCv2BmWb2aNRthR6RJSKyj2o2Kbj7fwGjgLuAS4HFZvZTMxsZc2wiIpJhaZ1TiK4IKo1eNUAP4HEzuyXG2EREJMOavfrIzK4GPgusB/4EfMvdq6M7jxcD3443RBGRPVNdXU1JSQmVlZXZDiVjCgsLKS4uJj8/f4/mT+eS1N7Ax919RfJId68zs7P3aK0iIhlQUlJCly5dGDZsGPvD6VB3p7y8nJKSEoYPH75Hy0in+WgGsKH+g5l1MbMjogAW7tFaRUQyoLKykl69eu0XCQHAzOjVq9de1YzSSQq/B7Ylfd4ejRMRafP2l4RQb2+/bzpJwZK7nnD3OmK+E1pEZF9QVFT0oXGLFi3ixBNPZOLEiYwdO5YrrriCZ599lokTJzJx4kSKiooYPXo0EydO5JJLLuGll17CzLjrrrsSy3j77bcxM2699dZWjzmdpLDMzK42s/zodQ2wrNUjERHZD1x99dVce+21zJ49m4ULF/LVr36V008/ndmzZzN79mwmT57MAw88wOzZs7nvvvsAOOigg3jkkUcSy3j44Yc55JBDYokvnaRwJXA08AGhO+wjgCtiiUZEZB+3Zs0aiouLE58POuigZucZMmQIlZWVrF27FnfnmWeeYcqUKbHE12wzkLuvI3R7LSLSbv3wyfksWL2lVZc5bmBXfnDO+BbNc+2113LSSSdx9NFHc9ppp3HZZZfRvXv3Zue74IILeOyxxzj00EOZNGkSHTp02NOwm5RO30eFZvYVM7vdzO6uf8USjYjIPu6yyy5j4cKFfOITn+Cll17iyCOPZOfOnc3O98lPfpLHHnuMhx56iGnTpsUWXzonjO8H3gVOB24CLiY8H0FEpN1o6RF9nAYOHMjll1/O5ZdfzoQJE5g3bx6HHXZYk/P079+f/Px8nnvuOX7961/z2muvxRJbOknhAHf/hJmd6+73mtmDhAfniIhICz3zzDOcfPLJ5OfnU1paSnl5OYMGDUpr3ptuuol169aRm5sbW3zpJIXq6H2TmU0g9H80LLaIRET2ERUVFbudVP76179OSUkJ11xzDYWFhQD8/Oc/p3///mkt7+ijj44lzmTW3NMvzezzwJ+Bg4B7gCLge+7+x9ijS2Hy5Mk+a9asbKxaRNqZhQsXMnbs2GyHkXGpvreZvenuk5ubt8maQtTp3RZ33wi8AozYm0BFRKRta/Lqo+ju5asyFIuIiGRZOjevPWdm3zSzwWbWs/4Ve2QiIpJx6Zxovjx6/0rSOEdNSSIi+5x07mjes065RUSk3UnnyWuXpBrv7ve1fjgiIpJN6ZxT+EjS6zjgRmBqjDGJiOwzSktLueiiixg5ciTjxo3jzDPP5L333sPM+O1vf5sod9VVV3HPPfcAcOmllzJo0KBE9xfr169n2LBhGYm32aTg7l9Nen0BOBQoiD80EZH2zd0577zzOPHEE1m6dCkLFizgpz/9KWvXrqVv3778+te/pqqqKuW8ubm53H135ruZS6em0FAFMKq1AxER2de8+OKL5Ofnc+WVVybGTZw4kcGDB9OnTx9OPvlk7r333pTzfu1rX+OXv/wlNTU1mQoXSO+cwpOEq40gJJFxwKNxBiUi0uqevg5K32ndZfY/CKb8rNHJzXV0d9111zFlyhQuv/zyD00bMmQIxx57LPfffz/nnHNOq4SbjnQuSU1+3lsNsMLdS2KKR0RkvzF8+HAOP/xwHnzwwZTTr7/+eqZOncpZZ52VsZjSSQorgTXuXglgZh3NbJi7L481MhGR1tTEEX1cxo8fz+OPP95kmeuvv54LLriA448//kPTDjjgACZOnMijj2aucSadcwqPAXVJn2ujcc0yszPMbJGZLTGz61JM/6WZzY5e75nZpvTCFhFp+0466SR27tzJnXfemRg3c+ZMVqxYkfg8ZswYxo0bx1NPPZVyGTfccAO33nprymlxSCcp5Ll74vR4NNzs1UdmlgvcBkwhnIeYZmbjksu4+7XuPtHdJwK/Bf7SkuBFRNoyM+Ovf/0rzz33HCNHjmT8+PHceOONDBw4cLdyN9xwAyUlqVvlx48fz6RJkzIRLpBe81GZmU119+kAZnYusD6N+Q4Hlrj7smi+h4FzgQWNlJ8G/CCN5YqItBsDBw5M2fwzb968xPAhhxxCXd2uBpn6+xXq/eUvmTteTicpXAk8YGa/iz6XACnvcm5gELAq6XMJcESqgmY2FBgO/KOR6VcAV0A4Iy8iIvFIp++jpcCRZlZEeCjP1jSXbakW10jZi4DH3b22kRjuAO6A8JCdNNcvIiIt1Ow5BTP7qZl1d/dt7r7VzHqY2Y/TWHYJMDjpczGwupGyFwEPpbFMERGJUTonmqe4e+KqoOgpbGemMd9MYJSZDTezAsKOf3rDQmY2GugB/Du9kEVE0tfcI4f3NXv7fdNJCrlm1qH+g5l1BDo0UR4Ad68hPLXtWWAh8Ki7zzezm8wsuUO9acDDvr/95UQkdoWFhZSXl+83icHdKS8vp7CwcI+Xkc6J5v8FXjCz/4k+Xwak7qyjAXefAcxoMO77DT7fmM6yRERaqri4mJKSEsrKyrIdSsYUFhZSXFy8x/Onc6L5FjObC5xCOHn8DDB0j9coIpIh+fn5DB+u54S1RLq9pJYS7mo+HziZ0BwkIiL7mEZrCmZ2IOHk8DSgHHiEcEnqRzMUm4iIZFhTzUfvAv8EznH3JQBmdm1GohIRkaxoqvnofEKz0YtmdqeZnUzqG9JERGQf0WhScPe/uvuFwBjgJeBaoJ+Z/d7MTstQfCIikkHpPKN5u7s/4O5nE+5Kng18qBtsERFp/1r0jGZ33+Duf3T3k+IKSEREsqdFSUFERPZtSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIgmxJgUzO8PMFpnZEjO7rpEynzSzBWY238wejDMeERFpWl5cCzazXOA24FSgBJhpZtPdfUFSmVHAd4Fj3H2jmfWNKx4REWlenDWFw4El7r7M3auAh4FzG5T5AnCbu28EcPd1McYjIiLNiDMpDAJWJX0uicYlOxA40Mz+ZWavm9kZMcYjIiLNiK35CLAU4zzF+kcBJwLFwD/NbIK7b9ptQWZXAFcADBkypPUjFRERIN6aQgkwOOlzMbA6RZm/uXu1u78PLCIkid24+x3uPtndJ/fp0ye2gEVE9ndxJoWZwCgzG25mBcBFwPQGZZ4APgpgZr0JzUnLYoxJRESaEFtScPca4CrgWWAh8Ki7zzezm8xsalTsWaDczBYALwLfcvfyuGISEZGmmXvDZv62bfLkyT5r1qxshyEi0q6Y2ZvuPrm5crqjWUREEpQUREQkQUlBREQSlBRERCRBSUFERBKUFEREJEFJQUREEpQUREQkQUlBREQS4uwlVUT2d3V1UF0BVdvB66DrgMzH4A6LZsBrv4MOXaD3KOh1QHj1HgVF/cBSdeq8f1JSENnf1dVB9Xaoqkh6j3bkVdt3DVdXNCgTvTdVpmbH7usacSKc8kMYODEz361sETxzHSz9B/QcATu3wPsvQ03lrjIdukKvkdBr1K6E0XsU9BwJBZ0yE2cboqQgsi+r2QklM+H9V2DVf6Byc4Md9/bdd5DpyMmDgs6Q3znsNPM7hc8de0C3QVBQFI3rtHuZyk3w79vhjhPgoE/ASf8FPYbF8rWp3Awv3wJv/CHEcMbP4COfh9z8kAS3lMD6xVC+ZNf7yn/DO4/uvpyuxdD7gJAw+oyGgy4I33Mfpg7xRNJRVQEr/gV9x0K34mxH07i6WlgzJxwNv/8KrPh3OFq3HOg3AYr67tqJp9pxF3ROvcNPLptXsOfxVW6GV38Fr/8e6mrCjvr4b0HnXq30/etgzoPw/I2wfT1M+gyc9H0oSvM5LFUVsGHphxNG+ZJQy+gyAKb+Fkad2jrxZlC6HeIpKYg0pa4O5j0edjJbPgjjeo6EESfA8BNg2HGtt0PbE+6w/j1Y9nJIBMv/GXa8AH3GhBhHnABDj4GO3bMXZ0NbVsOLP4XZD4SaxTG53q2RAAAP/ElEQVTXwJFf3rvmmpJZMONbsPotKD4cptwMgya1TrzuYblPfAXKFsKhn4HTfwKF3Vpn+RmgpCCyt1a+Ds98N+wMBkyE478Jm1aGHfCKf0HVNsCg/4Ro53siDDkKOhTFG9emlaEWsCyqDWwrDeO7DwlxDD8Bhh8PXfrFG0drWPcuvPDDcCK4ywA48bsw8WLIbUHL9ta1YRmzHwgnjU+9CQ76JOTEcHFlzU546Wfwr19Bl4Fw7m9h5Emtv54YKClI++YOOzbCphWwcTlsXBGq72POgoGT4r1aZONyeO4HsOCJsKM6+Qdw8IW772Rqq2H127uO0Fe9AbVVob190OSoJnE8FH8E8jqkv2532LkVtpeF5o/t63YNb1oJy1+Fje+Hsp37hHXUJ4Gew1t1M2TUitfgue+H8x+9R8MpN8LoKU3/nWuq4D9/hJduDudFjvpyaIrq0CX+eEvehCeuDLW0wy6D034U/3rr6gCHnNw9ml1JQdq+6h1hR7dxRdLOf3k0HCWBZJYTLmvsNwEmXRJOVnbq2XrxVG6Gf/4itHfn5IUmjaO/GtrU0/kuK1/f1Za/+u0Qa15HGHpU2GkPPiKU274+2tGvSxoug23Re+3O1Ovo2DMso77pqu/YfetSSndY+GQ46i9fEmpdp94Egw//cNklz8PT10H5Yhh1Gpz+/8IJ4Uyq3gEv/iRc6tptMJz7u/C3aW2bP4DZD8Lb94XvOfbsPVqMkoK0LRUbYOafwj97/ZF/fbNHvbxC6D4UegyN3oclDQ8NZeb9Gd66L+x0czvAuKkhQQw9ds+bC2pr4K17Qxt3RTlM/FS4MqbrwD3/vjs2hSam+maesoUfLpNbEI72d3v1Du9FfXcNd+4DnXrv3Qne9qS2OvyNX/pZSJxjzwm1td6jYMMyePaG0NzUc0S4qujA07Mb78o34IkvhRPUH/kCnPrD9A4kmlJbDYv/Dm/eC0ueCwcYw4+H478Nw4/bo0UqKUjbsWMT3DcV1swNR1TJO/oew3YNt+QmojVz4e37Ye4j4Qi/x/Bwpckhn2rZDVJLnodn/yvstIceA6f/NJ5r6LeuDVcFdeiya4ffoeu+daTf2nZug9dvh3/9OhyVjzo13G+QWxCaiY78Usua5uJUVQH/+FGoZfYYCh/7PQw9uuXLKV8aEuKch2DbWijqD4deDId+OiTBvaCkIG3Dzm1w/3nhyH7aQ61/KV/1jtDk8NZ94cobyw1HjpMugQNObfyE5bp34e83hKTQY3hoEx5ztnbSbdG2MnjlltCEMuascPNbNu6MTsfyf8Hfvhxqwkd+CU76XvNXVFXvgAXTw294xavp/4ZbSElBsq96BzzwiXAS8ZP3hmaAOJUvDbWH2Q82fpS1fX1oJnrznnAp5AnfhsOv2H+aZiR+O7eFS5hn3hnujv7Y71OfF1kzNySCdx7du9pumpQUJLtqdsLDn4IlL8D5fwp3gmZKfXvsW/eF9/r22EGHwcy7wl28H/kcnHBddu8xkH3bspfhb1eFu6ePugo+ekO4iOCdx8Nvc83s1jsvlgYlhbZm7fxwy331jnDpYm1N9F4V7uysrQo7s9pqqKtOXQYL1553HQhdB4XLJeuH69879Yr1h5WW2mp47FJ49ymY+rtw9JMtW1aH69ffuj9c1TTqdDjtx9DnwOzFJPuPyi3w3PdCzbTbkHB1Wc0O6DseDvts619B1wQlhbZkxb/hwQvDEWtRn3CiLCc/9MOSmx99zgvvu41rUKauNlyxs2V1eG1dEyWLJDn5oepZnyi6JA33HgX9xsf7Xetq4S9fCFcJTfk5HHFFvOtLV11duJKlS/9sRyL7oyUvwMs3h8uIJ10S/702KaSbFNQhXtwWPw+PfDp0FPaZJ6D74NZbdl1dOPLY8kFSoli9a3j127Blxu49VR58YbiML46jk7o6mH51SAin3tR2EgKE2pMSgmTLASeHVzugpBCneX+Bv1wRjg4+/Zf0O+VKV05OaE7q0q/xPl7q7wzeshoWTg83Zy15Ac78OYw/r/WOVtzh6W/D7P8NXRUcc03rLFdEMkpPXovLm/fA45dD8WS49KnWTwjpMgu1gv4T4KPXwxdfCbWVxy8LNZgta/Z+He6h3XTmnXD01XDCd/Z+mSKSFUoKcXj1V/DkNXDAKaGG0JZ6Uuw3Hj73PJz6o3CN/m1HhJOwe3Nu6aWfwWu/je7mvEnX+ou0Y0oKrck9XJ/8/A9g/Mfhogfb5pObcvPgmKvhS6+FGsT0q8INZhtXtHxZr/4KXv5ZuBdgyi1KCCLtnJJCa6mrhf/7Orz6y9Br4vl/avs3RPUaCZ99Cs7679AX/e1Hwet/CN8lHW/cERLghAvgnN9k/1JYEdlrsf4Xm9kZZrbIzJaY2XUppl9qZmVmNjt6fT7OeGJTWx0uw5x1Nxx7LZz9yz3u3jbjcnLCjVxfeT301fLMd+B/poRn2zblrfvg6W+FriHO+0P7+b4i0qTYkoKZ5QK3AVOAccA0MxuXougj7j4xev0prnhiU1UR7tyd9+fQB/wpN7bPJpRuxXDxY3DeHaGP+D8cC6/cGhJeQ3MfC5eeHnAKXHB3uI9CRPYJcdYUDgeWuPsyd68CHgbOjXF9mVe5Gf73fFj8HJz9q1BLaM/M4JAL4Sv/CR2P/eNHcOdHQ++e9RZMh79+EYYdCxf+b9vppVJEWkWcSWEQsCrpc0k0rqHzzWyumT1uZinv7DKzK8xslpnNKisriyPWltu+Hu45Ozwp6oK7YPJl2Y6o9RT1hU/cAxc+ANvWwR0fhed/GHojffzy0IfQtIchv2O2IxWRVhbnzWup2lAaXvf4JPCQu+80syuBe4EPPfDU3e8A7oDQzUVrB9pim1aFq3U2l4Sd46hTsh1RPMaeDcOOCc8bePW/w7gBh8CnH4//OcQikhVxJoUSIPnIvxhYnVzA3cuTPt4J3BxjPK1j/RK479zwqMjP/DU8anFf1rEHfOw2OOh8mP9EeAJWW7rvQkRaVZxJYSYwysyGAx8AFwGfSi5gZgPcvf6W2qlAimcWtiFr5sD9Hw/Dlz4Vjpr3FyNPCi8R2afFlhTcvcbMrgKeBXKBu919vpndBMxy9+nA1WY2FagBNgCXxhUP7/4fzHl475ax7OXwOMVL/pb5h4SLiGRArB3iufsMYEaDcd9PGv4u8N04Y0jYsRHWL967ZQyaBOf+Lly+KSKyD9p/ekk99NPhJSIijVK/BCIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikmDu2X+6ZUuYWRmwIttxNKI3sD7bQTRB8e2dth4ftP0YFd/e2Zv4hrp7n+YKtbuk0JaZ2Sx3n5ztOBqj+PZOW48P2n6Mim/vZCI+NR+JiEiCkoKIiCQoKbSuO7IdQDMU395p6/FB249R8e2d2OPTOQUREUlQTUFERBKUFEREJEFJoYXMbLCZvWhmC81svpldk6LMiWa22cxmR6/vZzjG5Wb2TrTuWSmmm5n9xsyWmNlcM5uUwdhGJ22X2Wa2xcy+1qBMxrefmd1tZuvMbF7SuJ5m9pyZLY7eezQy72ejMovN7LMZiu3nZvZu9Pf7q5l1b2TeJn8LMcd4o5l9kPR3PLORec8ws0XR7/G6DMb3SFJsy81sdiPzxroNG9unZO335+56teAFDAAmRcNdgPeAcQ3KnAg8lcUYlwO9m5h+JvA0YMCRwBtZijMXKCXcVJPV7QccD0wC5iWNuwW4Lhq+Drg5xXw9gWXRe49ouEcGYjsNyIuGb04VWzq/hZhjvBH4Zhq/gaXACKAAmNPw/ymu+BpM/wXw/Wxsw8b2Kdn6/amm0ELuvsbd34qGtwILgUHZjarFzgXu8+B1oLuZDchCHCcDS90963eou/srwIYGo88F7o2G7wU+lmLW04Hn3H2Du28EngPOiDs2d/+7u9dEH18HiltznS3VyPZLx+HAEndf5u5VwMOE7d6qmorPzAz4JPBQa683HU3sU7Ly+1NS2AtmNgw4FHgjxeSjzGyOmT1tZuMzGhg48Hcze9PMrkgxfRCwKulzCdlJbBfR+D9iNrdfvX7uvgbCPy7QN0WZtrAtLyfU/FJp7rcQt6uiJq67G2n+aAvb7zhgrbsvbmR6xrZhg31KVn5/Sgp7yMyKgD8DX3P3LQ0mv0VoEjkE+C3wRIbDO8bdJwFTgK+Y2fENpluKeTJ6bbKZFQBTgcdSTM729muJrG5LM7sBqAEeaKRIc7+FOP0eGAlMBNYQmmgayvpvEZhG07WEjGzDZvYpjc6WYtxebT8lhT1gZvmEP94D7v6XhtPdfYu7b4uGZwD5ZtY7U/G5++rofR3wV0IVPVkJMDjpczGwOjPRJUwB3nL3tQ0nZHv7JVlb36wWva9LUSZr2zI6qXg2cLFHDcwNpfFbiI27r3X3WnevA+5sZN1Z/S2aWR7wceCRxspkYhs2sk/Jyu9PSaGFovbHu4CF7v7fjZTpH5XDzA4nbOfyDMXX2cy61A8TTkjOa1BsOnBJdBXSkcDm+mpqBjV6dJbN7dfAdKD+ao7PAn9LUeZZ4DQz6xE1j5wWjYuVmZ0BfAeY6u4VjZRJ57cQZ4zJ56nOa2TdM4FRZjY8qj1eRNjumXIK8K67l6SamIlt2MQ+JTu/v7jOqO+rL+BYQvVsLjA7ep0JXAlcGZW5CphPuJLideDoDMY3IlrvnCiGG6LxyfEZcBvhqo93gMkZ3oadCDv5bknjsrr9CAlqDVBNOPr6HNALeAFYHL33jMpOBv6UNO/lwJLodVmGYltCaEuu/w3+ISo7EJjR1G8hg9vv/uj3NZewgxvQMMbo85mEK26WxhVjqvii8ffU/+6SymZ0GzaxT8nK70/dXIiISIKaj0REJEFJQUREEpQUREQkQUlBREQSlBRERCRBSUGkATOrtd17cm21njvNbFhyT50ibU1etgMQaYN2uPvEbAchkg2qKYikKepX/2Yz+0/0OiAaP9TMXog6fnvBzIZE4/tZeNbBnOh1dLSoXDO7M+o7/+9m1jFrX0qkASUFkQ/r2KD56MKkaVvc/XDgd8CvonG/I3RFfjChY7rfRON/A7zsoWO/SYQ7YgFGAbe5+3hgE3B+zN9HJG26o1mkATPb5u5FKcYvB05y92VRB2al7t7LzNYTunCojsavcffeZlYGFLv7zqRlDCP0fz8q+vwdIN/dfxz/NxNpnmoKIi3jjQw3ViaVnUnDtejcnrQhSgoiLXNh0vu/o+HXCL17AlwMvBoNvwB8CcDMcs2sa6aCFNlTOkIR+bCOtvtD3J9x9/rLUjuY2RuEA6pp0birgbvN7FtAGXBZNP4a4A4z+xyhRvAlQk+dIm2WzimIpCk6pzDZ3ddnOxaRuKj5SEREElRTEBGRBNUUREQkQUlBREQSlBRERCRBSUFERBKUFEREJOH/A8oP3lme+QHeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2692922b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Testing Accuracy\n",
    "x = np.arange(20)+1\n",
    "\n",
    "plt.plot(x, model_hist_embedding.history['val_acc'])\n",
    "plt.plot(x, model_hist_cnn.history['val_acc'])\n",
    "plt.legend(['LSTM', 'CNN'], loc='right')\n",
    "plt.ylabel(\"Accuracy\")\n",
    "axes = plt.gca()\n",
    "axes.set_ylim([0.45,1.01])\n",
    "plt.xlabel(\"Epoch\")\n",
    "plt.title(\"Testing Accuracy\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So Basically they are all overfitting!\n",
    "This is a small dataset to show what can be done with deep learning"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
